Why correcting the scientific record is hard
Here I explain, from a very simple example at a modest scale, why correcting the scientific record is difficult in today's academic world. I think it highlights, in a very simple way, some flaws of our systems that we can hopefully address. (Also worth noting: although the example comes from the condensed matter field, you don't have to know anything about physics to read the text below!)
I have recently worked, with a MSc student in my group, on the understanding of elastic behavior of porous materials. During the course of his internship, we spent a lot of time in understanding exactly how elastic stability criteria work in the most generic case. Stability conditions, know as Born stability criteria, have very simple and well-known mathematical expressions for cubic crystals, but things get more complex for systems with lower symmetry. While the fundamentals have been set forth very clearly a long time ago by Max Born (in particular in his 1954 book, Dynamics Theory of Crystal Lattices), his and subsequent books usual give explicit expressions only for common high-symmetry crystals (cubic, hexagonal, etc.)
Now, during the course of the MSc project, we stumbled onto some papers that quote incorrect mathematical formulations of the Born conditions, while usually citing the original Born book (in which these expressions are not found). Most of the errors arise from people incorrectly generalizing the "cubic" conditions. We looked a bit more, and found more examples of such errors, in papers between 2007 and 2014. Now, if you find several mistakes in series of related equations, in a dozen papers published in a given field throughout a decade, what do you do?
Over the course of a few days, we wrote a short paper, explaining the general form of the conditions, and explicitly listing their mathematical expression for each possible crystal class. We uploaded it to the arXiv, and sent it for publication to a well-read journal in the field (Phys. Rev. B). I highlighted, in the cover letter to the editor (see below), that it was meant to be a concise and pedagogical reference that would benefit the research community, and it seemed to us worthy of publication.
First, it took the editor some time to send the paper out for review. He first sought input from a member of the editorial board, we were told. My cover letter was apparently convincing, as the manuscript was sent for review and we got three referee reports on this. I won't quote them in full (I don't have their permission), but they can be summarized as:
- After a considerable amount of thinking, this is appropriate for publication in PRB. The multiple erroneous references cited show it is valuable, though it shouldn't be needed in an ideal world. [Then some useful comments for improving the work. Thanks!]
- The basic principles are well-established, the derivations are mathematically simple. Reject.
- I’m in favor of publishing the paper; whether in PRB is an editorial decision best left to the PRB editors. [Also known as the “no risk” approach to refereeing ;-)]
Editor decided paper couldn't be published, but we could resubmit if we felt we had a strong response. It underwent further review, during which one of the referees said something very interesting:
I have checked "Enough significant new physics? No" both times, but still tend to agree with the third referee -- this should be published somewhere
I think this is interesting, because it shows why correcting the scientific record is hard: it is considered not new science. The current standards for publication highlight (with some good reason) the original and the sexy. But while doing so, we need to keep room for corrections, comments and allow the discussion to go on, through peer-reviewed articles.
Another interesting point, made by the reviewer, is the unstated comment that maybe PRB is too good a journal for this sort of work. I think it also shows that corrections and pedagogical papers are deemed less important than your “regular” research article.
Finally, in our case, the paper was accepted and is now published. We managed to convince the reviewers and editor of the value of our paper to the community, which should be in my eye an important yardstick for publication.
Let me know your thoughts, in comments or on twitter. Also: what do you think should come next? Contacting the authors? Post-publication peer review?