Tuesday, August 26, 2008

When the refereed becomes the referee...

Last week I talked about how much fun I was having (NOT!) responding to a peer review of one of my papers. Well, now the tables are turned, and I am working on a review of someone else's paper.

It's a bit nerve-wracking. I am supposed to look over the science and make judgements on the quality of another person's work. But more than that, the author usually knows more about what she/he is writing about than I do. So, if I want to do a good job as a peer reviewer, I need to read not only the paper that I'm reviewing, but a lot of background literature.

There are lots of things that need to be considered in a peer review. These include:

  • Is the research new and useful? It doesn't need to be the most exciting astronomy ever, but if a paper has little use to anybody else, then the big astronomy journals tend not to want it.
  • Is the research convincing? Do I believe the results? Even if I don't believe the conclusions, do I at least believe that the data and the methods are sound?
  • Have the authors references enough of the literature? When we write a paper, we don't need to re-derive all of 500 years of astronomical research. It is okay to use the results of other people (like Newton did in his famous quote about "standing on the shoulders of giants."), but we have to give those people credit. And, as a referee, I need to make sure that the paper is not based on old results that have been discredited.
  • Did the authors do the math right? Astronomy involves a lot of math. If I were to be a super referee, I would re-calculate their numbers and re-derive their equations to make sure I get the same answers. I tend not to do that. Instead, I spot check numbers and see if the results make sense. This is not an ideal method! I'm being a bit lazy here, and I am giving the referee the benefit of the doubt, assuming that they've checked their own work for mistakes. But, generally, the only way to truly check the work would be for me to re-do their project. That would take time, and delay the publication of the paper. If I suspect a problem, it is okay for me to tell the authors that I think there may be an error, and for them to check the results.
  • Does the organization of the paper make sense?
  • Do tables and graphs and figures convey the information the authors claim, and do so in a clear manner?

In the end, as a reviewer I get large sway over whether or not a paper eventually gets published. If I reject a paper, the authors can ask the editor of the journal to send the paper to another reviewer, and that decision lies with the editor. If I say a paper is fine, it will almost always appear in print, but the editor would have the power to request changes or even send it to another reviewer, if he/she had reason to doubt the quality of my report. So far, I have never suggested that a paper be rejected.

I do feel a little bad for the people who get my reports. I tend to be long-winded, ask lots of questions, and make lots of suggestions. I just hope that the authors on the receiving end of my reports find my long litany of comments useful. After all, it is science that needs to be served, not my whims or the authors' own desires.

No comments:

Post a Comment