who will watch the watchmen?

Thanks to my dad for sending me an excellent article.

For concreteness, the article in question is a popularized discussion of a paper published in PLos Medicine (a high profile medical journal) entitled Why most published research findings are false.

I think that Ioannidis (the author of the PLoS paper) makes some excellent points, but I am more confident in the quality of scientific publications than is he. Should you agree with me? I'll let you judge for yourself, once you understand the idea behind the argument.

So here's the basic idea. As a scientist, you impress funding agencies and hiring committees (and secure yourself a career), at least in part, by publishing in highly selective journals. Those journals only want to publish results that are "surprising" in some way. Now, on their own, both of these things are completely reasonable.

However, combining these properties, you get the result is that surprising work is more often published, and has more impact in the scientific community than does less surprising work. Here's a quick example (from Ioannidis, quoted by Freedman in the Atlantic article) to show how this works, which I modify slightly for my purposes.

The results of most experiments have some intrinsic randomness associated with them. So, if you repeat the experiment a few times, you expect to get a different, but (probably) similar, result each time. If you repeat the experiment enough times, you eventually get an answer that is very different from the norm. If you are repeating the experiment yourself, you know this, and identify the unusual result as being a statistical fluke. When you report your result, you include all of the trials (or even omit the outlier), and the reader has a good knowledge of the typical result, and the variation they can expect. This is good science, and is not a problem.

Now imagine that the experiment is very long and costly to perform, so you only do it once. With high probability, you get the typical (maybe boring) result, and either publish it in a low-ranking journal (where not many people read it), or not at all. However, there is some (maybe small) chance that you will discover something exciting, and will not know that the result is atypical, inasmuch as the result would not occur often if the experiment were repeated many times. If you do get the "exciting" (surprising) result, you publish it in a high-profile journal. Here, as in the first example, you are still not doing anything "wrong" as a scientist. Since the experiment can't be repeated, you can't say if the result is typical or not, but that's how it goes.You just report thoroughly what you observed, and how you did the measurement, and any relevant interpretations you made, and leave it to your readers to make responsible use of your results.

But, to save time in wading through the mountains of work being published, most scientists (myself included) start by reading the "important" journals, and don't spend as much time digging through the lesser ones.

Interestingly, the end result for the community seems to be that statistically atypical results have more prominence than do more typical ones. And no one has to do anything overtly "wrong" for it to happen: it's a natural consequence of giving more exposure to more "surprising" research.

So that's Ioannidis' argument, and it's pretty compelling.

As a theorist, I like to imagine that I am immune from such things (since I don't really do experiments, these "randomness" effects from experimentation don't affect my work in the same way). However, when I sit down to formulate new theories, they are often heavily guided by the observations of experimenters. And I, too, spend more time reading high-profile papers than low-profile ones. So, in some sense, I am as vulnerable as anyone else.

What to do about this? Well, I think we, as a scientific community, should be more prone to publish negative results (ie: "I didn't see anything interesting happen"), as well as positive ones (ie: "OMG! It totally turned blue!", or whatever). We should probably also not put such a premium on papers from high profile journals, especially in terms of what we read to direct our research.

So, this is my mid-October resolution: I will spend more time reading results from low-profile journals, and give those results the same amount of thought that I put into higher-profile ones.