Is speculation a sin, or a stimulus?

It’s a decision that comes at the end of every presentation, every paper, and in some ways, every experiment. You’re at the limit of established knowledge, having followed the path of learning from its highways, to its paths, to its narrow winding tracks, and now you find yourself inching along a thin curving ridge with the precipice of the unknown yawning beyond and beneath. Each bit of data is a step, a handhold, something to give an increment of progress as you venture further into the void. And as you cling there, you face a choice – do you stick close to the wall, tremulously seeking out new purchase, or do you lunge around the corner but in so doing, risk losing your grip?

We’re taught what’s good conduct. Cautious and tight interpretation of our own data. Results placed in the context of previous work, corollaries noted and contradictions addressed. Not to look further than the next handhold. Save your strength for the way ahead. But all the same…that itch, that compulsion that brought you here in the first place, that drive to confront the abyss, that desire to go beyond what’s been done before. You have a sense of what’s around the corner already. Why not gamble on your own intuition and logic?

It’s a choice, and a temptation, that divides us. Some would contend that in speculating what might be ahead, you generate hypotheses that you and others can test. You stimulate yourself and the community that shares that crag with you. Better to speculate and be wrong, the thinking goes, than to offer nothing at all. You may lose your way, you may even fall completely, but as long as you flag your actions clearly, that you let others know you’re lunging, taking a leap of inspiration and faith, then it’s only yourself that’s at risk.

Other heads would disagree. No, they shudder, it’s unconscionable. You’re mistaking a finger-grip for a handhold. Your footing isn’t secure. And worse, by heading that way you risk not just yourself, but your friends, your colleagues, and those who funded your expedition. You’re imperilling the whole thing. You’ve lost your mind.

How far from the data is it acceptable to go then? Perhaps the key thing is if your speculation makes predictions that can be easily tested – reaching around the corner for a handhold that you think is there, but not leaping blindly ahead and trusting that there will be something to catch onto. Careless talk can cost careers, and signposted speculation is very, very different from loose interpretation of your own data. Overinterpretation – claiming that you’ve shown something you haven’t – is the way to ruin. If you mistakenly or optimistically conclude something that simply isn’t supported by your data, then you might be taking the first step off the edge. It’s a reminder of the importance of experimental rigour, of proper technique. An expert can take a calculated risk; a novice can’t tell the difference between a calculated risk and a near-certain plunge into the abyss – and when they try, they endanger not just themselves but others. Speculation and imprecise reasoning are two different things.

Mirages. Phantom handholds. Ghost ledges. That’s what’s created by overinterpretation. Conclusions that go beyond what the data have shown, that get repeated without further verification (easy if you cite papers you haven’t read carefully!), that become accepted as firm and solid despite never being directly tested. That actually aren’t true, and don’t exist. That at some point will begin to lose their solidity and crumble and disintegrate as their insubstantial nature becomes clear. How long before the whole route goes crashing into the chasm, and how many people will be on it when it does?

In philosophical terms, should a hypothesis come only at the beginning of a piece of work, or can it come also at the end? It has to come at the beginning (“This is the way to the summit!”), but once you’ve completed a stage and are resting, plotting the next move, is it ok to take a guess at what’s ahead? Or should you stick instead to grasping the next handhold and shuffling on?

So why do it? In some senses, it’s a way of establishing conceptual priority, taking credit for an idea that you haven’t yet tested. But it’s also – and this should not be sneered at – fun. It’s creative. It’s about synthesising something, trying to see the way through the snow and the ice and the blizzards. And if we only inch along, perhaps we’ll never travel fast enough.

There’s a great exchange in George MacDonald Fraser’s marvellous Flashman series in which the titular antihero challenges Abraham Lincoln with a riddle. Why, he asks, do you say it is right that the American colonies declared their independence from Britain, but wrong for the Southern states to secede from the Union? Ah, Lincoln replies, that’s simple – the War of Independence was right because it was successful, while the Southern rebellion was wrong because it failed. Perhaps speculation is the same. It’s ok if it turns out to be right, but cause for censure if it’s wrong. But even if it’s wrong – so long as nobody is misled – perhaps it can ultimately bring us closer to the truth? The way is out there – we just have to find it.