03 January 2011

Sciency Unreliability

Winston Churchill once famously said “Democracy is the worst form of government except for all those others that have been tried.” I think the same could be said of science.

I was thinking about this after my recent “Sciency Answer” about variegated plants. After I posted it, I had a couple of conversations with Kelly Norris on the topic. He read the studies I cited to support my explanation, and wasn't impressed with their technique and reasoning. I (obviously) was. The upshot is that I'm pretty confident variegation in wild caladiums (and other plants) is a disguise to prevent insect damage, and Kelly is very skeptical. Which is totally normal. Scientists often disagree about what studies mean, because scientists know a very important thing about science: It is often wrong.

Since I'm now in the business of dispensing sciency answers, I thought I should talk about that. About why science is so often wrong, why some of my answers may prove to be wrong. There are lots of reasons, but I think one of the biggest problems is people like me -- Graduate students.

Graduate students like myself are the people in the lab (or field or hospital) doing the actual work of most of the scientific research going on these days, and we have a HUGE conflict of interest that effects any sort of research we do. We desperately want to graduate.

In order get my PhD, I do research. In my case, research on petunias. I perform experiments, write up my findings as a dissertation, and hey presto, you have to call me Doctor. Unless, of course, my research doesn't work and I don't find anything interesting. To use an extreme, completely made up example: if I was studying the effects of chewing gum, and found that it caused cancer, WOW! That's shocking! It gets published in a fancy journal, I get a degree, I get a job, and everything is wonderful. But if chewing gum doesn't have any effect at all... I'm screwed. No chance of a good publication, no job, and maybe even no degree unless I start over with a new line of research.

This isn't just true for graduate students. University faculty need to make tenure, they need grants, and to get all that, they need publications. Publications from research with big, interesting findings.

In response to all the pressure to find results, people do sometimes make stuff up, but most of the time they do something more subtle, perhaps even unconscious. They overlook alternative explanations, massage their data, or keep asking the same question a different way until they find something that passes the test of statistical significance. Chewing gum may not cause cancer but it must do something... gum disease? Jaw injury? Stress? Divorce rate? Ask enough questions, and even if just by chance, the numbers will tell you one is right.

Because of that, new scientific findings tend to overstate the case – they find big, dramatic effects that sometimes prove to be weaker, or nonexistent in future studies. Science does however, eventually, tend to correct its own mistakes. Since no one currently believes chewing gum is dangerous, a study finding it to be safe is boring and unpublishable. But if someone else had said it did cause cancer, debunking that finding would be very publishable. That's why in significant areas of research we get dueling studies (Eggs are good for you! Eggs are bad for you! No, they're good for you!) but over time, eventually, we can look back over all the studies, compare them, and finally (hopefully) come to a conclusions that is close to actual reality.

That is why, despite all its flaws, I believe in science as a powerful way to understand how the world really works. Just don't confuse science with the actual truth. Truth is something we strive for, but can never really, absolutely, know.


Kat said...

Great blog. I think too often we do confuse science with the absolute truth. Everything is open to interpretation and the more we know, the more the interpretation can change. What I am happy about is that science is infiltrating the world of everyday gardeners more and more. Gardeners no longer want to rely on just old-fashioned garden wisdom (although it can be good)alone. So I hope you'll keep posting all those sciency things.

Susan in the Pink Hat said...

The journals that publish these papers are supposed to act as a filtering device for this reason. If you are reading a paper in 'Nature' you can be sure that the person's methodology and theory has been rigorously examined by peer reviewers who have looked carefully enough to merit whether their theory holds water. If anything is wrong with empirical science, it's probably a lack of willingness to reproduce experiments to hold water. Why are graduate students trying to reproduce experiments or test other theories that have been floated for their thesis rather than trying to come up with something new?

Greensparrow said...

Kat, oh for sure! There is lots of great, reliable science about growing plants that hasn't made it to the general gardening public, and I'll keep talking about it!

Susan, the peer-review process is excellent, but sadly often fails, even in big name journals like Nature and Science. Many of them have been redacted or disproved. (see, for example, the recent kerflufle over bacteria living with arsenic http://www.nytimes.com/2010/12/14/science/14arsenic.html?_r=1)

And: Really new research is risky -- more chance of failure, and the resulting loss of career prospects, etc. Like any field, most people take the safe, easy route that guarantees moderate but unexciting success, while a few do bold, dramatic work that sometimes makes the famous and sometimes ruins them.

allanbecker-gardenguru said...

As usual, a wonderful, informative post. Oops! Wonderful is not a very sciencey word to describe such an erudite piece.
When do you expect to complete your PhD. and what do you plan to do after

Greensparrow said...

Allan, thank you! And I'll take wonderful. It is one of my favorite adjectives. The plan is to graduate this coming August (FINALLY!) and after that... well, we'll see. Various options are in the works. I'll talk about them on the blog as they unfold.

NotSoAngryRedHead said...

Well, one funny thing about the peer review process is that it's often done by graduate students and their professors sign off on it and send it in. That doesn't mean graduate students are incapable of reviewing an article for publication, but it's problematic nonetheless.

My big problem with scientific research probably boils down to publishing and the need for an effect or correlation when it's just as useful to scientists and researchers to see studies that showed no effect. There are stacks and stacks of unpublished research with no effect, but they're stuffed in drawers where no search engine can reach. Sure, there could be no correlation because there was a problem with the methodology, but there's also the chance that there simply wasn't a correlation.

For some research, you might actually want there to be no effect, but you won't be able to publish until someone finds one. Gathering research for meta-analysis is just mind boggling to me because I know there are probably loads more studies not included because they weren't published and the researchers could only gather a few of the unpublished studies based on leads they had.

I suppose it boils down to a lack of communication. I'm personally more interested in the similarities rather than the differences, but research is geared towards differences. I think people in general are innately geared to see differences rather than similarities.

For example, a lot of research is geared towards finding differences between men and women which helps support sexism. The media picks up this difference-finding research and feeds it to people, and AHA! there's a reason to not hire men into childcare positions and women into the military. It's only years later when someone publishes a meta-analysis study and finds no difference between men and women and their nurturing capabilities (a characteristic used to prevent men from becoming childcare providers and women from joining the military), but the problem is that it's been years and a meta-analysis isn't exactly sexy and easy to understand. So the media picks up on the differences that the study did support and reports on that instead. Huzzah!

Dr. Meston and Dr. Buss published a very sexy study that quickly spread through the international media about the reasons why people have sex. Of course, the similarities weren't reported in the news, but by golly, those differences were headline minus those messy t-values that get in the way. But what if they had found no difference?

Besides, what research manages to escape methodological/analytic/interpretation critique? Even when it comes to 2+2=4, there's still criticism.

Liz said...

I did some research as an undergrad, and without any need to get tenure or a degree I realized how faulty the research I was doing was. But even faulty research can get published..

Tom said...

But *doing* science is just so much fun! Actually that's not true, I just liked working in the greenhouses, I never actually liked the WORK I was doing. It is see someone from science admitting that science isn't infallible. That always kind of bugged me in school.

Anonymous said...

Science always has mistakes. The problem is when science or scientists are put on a elevated stand and given the status of gods.

Greensparrow do your research with due care and gusto. Write your chapters as you progress and enjoy the process. This is to say don't sweat what you will do after graduation. Most of my friends, except those in education, are not in a field normally associated with their studies.

You could always develop the German style trailing geranium to rival the trendy wave petunias!