Thursday, August 21, 2008

The referee report

In addition to my travels last weekend, another reason that I've been so quiet is because I've been working on revisions to a paper that I have submitted for publication in one of our professional journals. It's been a very intense process, and dries up all of my writing skills for the day.

In astronomy (and every science), when we submit a paper to a professional journal, it has to go through a process called "peer review," or "refereeing." The draft of the paper is sent to one or more experts in the field. These experts, who are expected take a fair and unbiased look at a paper, are given three options: (1) approve the paper, (2) send the paper back for revisions, or (3) reject the paper.

In astronomy, option (2) is by far the most common, especially if it is the first time the paper has been sent in. Very few papers are approved without any revisions. And outright rejection is rare, with only a few percent of papers never being published.

So, when I submit a paper, I usually wait a month and then get an email from the editor of the journal with the referee's report, a list of suggestions, questions, and criticisms. I am then expected to make a good faith effort to address the referee's report in my revisions. After I revise the report, I re-submit the paper to the journal, who then passes it on to the referee (almost always the same person as before). In most cases where the authors have made a good effort, the referee will accept the paper, often contingent on a few minor edits. Sometimes the referee decides it needs some more work and sends it back for more substantial revisions. And, rarely, the referee will decide the paper is just not up to snuff and reject it.

This process is designed to try and ensure that all papers published in a professional journal are scientifically rigorous and meet certain standards of quality. At least, this is the theory. The peer review process is not perfect. Sometimes bad articles get through. Sometimes the referee tries to hold up a good article for purely personal reasons. Sometimes a different or additional referee has to be brought in to resolve conflicts. But while the process isn't perfect, it generally works pretty well.

I've found that most referee's reports I get can be ranked on three scales: length, positive/negative, and helpfulness. Some reports I've received have been short and positive with a couple of useful suggestions. I've gotten reports that are short and positive, and only later did I find out that they weren't useful, because both the referee and I missed a glaring problem. I've gotten long, thoughtful reports that have really helped a paper. And I've gotten long reports that are fairly critical, with the criticisms making me wonder if the person actually read the paper. (As a fictional example, the referee's report might include the comment, "I am shocked that the authors did not discuss the price of tea in China," when the paper has a section headlined, "Our star and the price of tea in China.")

These latter types of reports are a pain. I feel I have to make at least one good-faith effort to address the report, but it is hard for me (a naturally sarcastic person) to avoid putting snarky remarks in my response. Once I had a long, negative report that, after a lot of thought, I realized came down to the referee misunderstanding a single word. I did a search and replace of that word in the paper, and the referee's report came back saying, "this paper is vastly improved." Alas, if they were all that simple.

As for my most recent paper, the revision process wasn't exactly fun. But I finished it, and, after my co-authors review the paper, I will be able to send it back in for another round with the referee.

No comments:

Post a Comment