Lessons from a failed experiment

I wanted high temp superconductors too!

I’ve been spending less of my time recently with larger teams, and more with some engineering around AI and LLMs. That’s been a good opportunity to exercise some first principles that I write about here. It’s easy to look at a situation in the past, with perfect insight, and have some good advice about it. Harder to live that advice in the moment.

The world got excited last month about the possibility of room-temperature super conductors. It turns out (it looks like now) that the initial paper was wrong, it’s some kind of magnetic effect, and there is no real super conductivity. Bummer!

When we were still in the middle of uncertainty, though, I observed some interesting things. I saw lots of folks falling back on things like reference to authority, to dismiss the finding. I work with someone who regularly spends time with very serious researchers, all of whom said, “no way, challenges how we view the world”.

They turned out to be right this time, but there plenty of examples of this kind of reaction being wrong. Most really radical paradigm shifts were rejected by the “old guard” for this kind of reason at first. There’s even a name for this, Plank’s principle, the idea that scientific consensus only changes through generational change.

Why is this? And why should we not just “listen to the experts”? In a word, incentives. The world values new discoveries. But science only gives credit for the ideas that work, not the novel attempts that don’t work. No one gets credit for lots of “well, that was interesting but it didn’t pan out” papers. We barely even give credit for replication studies and negative results! (though with things like preregistration, that’s getting a bit better now). And academic science has even worse incentives - scientists have to play politics and fit in to get tenure, funding, etc. It’s really hard for radical ideas to get traction.

But of course, as I’ve written before, it’s much better to have a default setting of “optimist” or at least “open minded and curious”. There is a small prize for being skeptical and right, no prize for being skeptical and wrong, and a huge prize for being open minded and right! (Ok - you can be open minded and wrong and waste a lot of time and money, so that square of the graph isn’t great, and you do want to have some proof points along the way as you scale up).

At any rate, I don’t feel any shame for trying to be open minded and rejecting the quick “appeal to authority” arguments, even though they turned out to be correct. If you want to break ground and do surprising things, you have to be comfortable with being wrong 20 times before you are right once. But the once matters, and makes it all worth it!