Sciency Unreliability

Winston Churchill once famously said “Democracy is the worst form of government except for all those others that have been tried.” I think the same could be said of science.

I was thinking about this after my recent “Sciency Answer” about variegated plants. After I posted it, I had a couple of conversations with Kelly Norris on the topic. He read the studies I cited to support my explanation, and wasn't impressed with their technique and reasoning. I (obviously) was. The upshot is that I'm pretty confident variegation in wild caladiums (and other plants) is a disguise to prevent insect damage, and Kelly is very skeptical. Which is totally normal. Scientists often disagree about what studies mean, because scientists know a very important thing about science: It is often wrong.

Since I'm now in the business of dispensing sciency answers, I thought I should talk about that. About why science is so often wrong, why some of my answers may prove to be wrong. There are lots of reasons, but I think one of the biggest problems is people like me -- Graduate students.

Graduate students like myself are the people in the lab (or field or hospital) doing the actual work of most of the scientific research going on these days, and we have a HUGE conflict of interest that effects any sort of research we do. We desperately want to graduate.

In order get my PhD, I do research. In my case, research on petunias. I perform experiments, write up my findings as a dissertation, and hey presto, you have to call me Doctor. Unless, of course, my research doesn't work and I don't find anything interesting. To use an extreme, completely made up example: if I was studying the effects of chewing gum, and found that it caused cancer, WOW! That's shocking! It gets published in a fancy journal, I get a degree, I get a job, and everything is wonderful. But if chewing gum doesn't have any effect at all... I'm screwed. No chance of a good publication, no job, and maybe even no degree unless I start over with a new line of research.

This isn't just true for graduate students. University faculty need to make tenure, they need grants, and to get all that, they need publications. Publications from research with big, interesting findings.

In response to all the pressure to find results, people do sometimes make stuff up, but most of the time they do something more subtle, perhaps even unconscious. They overlook alternative explanations, massage their data, or keep asking the same question a different way until they find something that passes the test of statistical significance. Chewing gum may not cause cancer but it must do something... gum disease? Jaw injury? Stress? Divorce rate? Ask enough questions, and even if just by chance, the numbers will tell you one is right.

Because of that, new scientific findings tend to overstate the case – they find big, dramatic effects that sometimes prove to be weaker, or nonexistent in future studies. Science does however, eventually, tend to correct its own mistakes. Since no one currently believes chewing gum is dangerous, a study finding it to be safe is boring and unpublishable. But if someone else had said it did cause cancer, debunking that finding would be very publishable. That's why in significant areas of research we get dueling studies (Eggs are good for you! Eggs are bad for you! No, they're good for you!) but over time, eventually, we can look back over all the studies, compare them, and finally (hopefully) come to a conclusions that is close to actual reality.

That is why, despite all its flaws, I believe in science as a powerful way to understand how the world really works. Just don't confuse science with the actual truth. Truth is something we strive for, but can never really, absolutely, know.