Tales From The Science Trenches: An open-letter to Frontiers in Human Neuroscience

Note: Over the next few months I'll be doing a series of posts critiquing modern peer-review process.  As part of this I'd like to collect stories from other scientists on their experiences trying to get papers through peer-review.  Good or bad, if you have a story you'd like to share, please email me (timothyv[at]gmail[dot]com) or put them in the comments section and I'll post them as I see them.  

In an ideal sense, the peer-review process is designed to provide useful critiques of scientific manuscripts so that the end result has minimal errors and justified conclusions.

Unfortunately, personal biases, poor editorial oversight and general abuse of the review process can sometimes impair communication of scientific results.  What is described below is one example of the flaws in the process that are sometimes experienced by those trying to publish scientific results.

Recently my colleagues and I tried submitting a manuscript to the journal Frontiers in Human Neuroscience.  Frontier is an open access set of journals that prides itself for a rapid turn-around (i.e., fast review cycles) and a simplified peer-review process focused primarily on methodological validity.  

The manuscript we submitted was a study on the brain activation changes that occur with increasing obesity.   The paper had already been bounced around to several other journals before being sent to Frontiers.  As sometimes happens, the same anonymous reviewer followed us from journal to journal providing the same general (and in our opinion incorrect) critiques of the manuscript.

Now for those of you outside the process, it's generally considered bad manners to agree to review a manuscript that you have already critiqued at another journal.  The logic being that you can't provide an unbiased review of the manuscript because you've already judged it previously.

Unfortunately, this particular reviewer keeps agreeing to critique the manuscript while letting his previous judgements color his critique of our manuscript.  What's worse is that he (I'm assuming the gender because I have a good idea who it is) isn't interested in making helpful criticisms that could expand or fix the paper.  You can see for yourself in the linked documents below.

So we submitted our work to Frontiers and this reviewer followed us there.  He gave his usual critique, ignoring many of the changes we that made to address these concerns.  We then spent a couple of months further revising the manuscript and running control analyses.   Once the revisions were submitted to the journal, this hostile referee immediately withdrew his review (again, rather than address our arguments) while the only other referee signed off on our changes as being sufficient for publication.

Then we waited 8 weeks with no word from Frontiers.  Again, for those of you non-scientists this is a pretty long time to wait for an editorial decision after the reviewers had submitted their responses.  Emails to the editor were ignored and the status remained unchanged.

It turns out that the original editor had retired after we resubmitted our manuscript.  Instead of seeing the submission through to completion (or not agreeing to be an editor in the first place), he simply withdrew as well.  It appears that it took Frontiers almost to figure this out.  Once a new editor was assigned, this individual (Dr. HH in the open letter below) made an executive decision to reject the manuscript based only on the withdrawn review.  He completely ignored our replies and made no mention as to why our carefully constructed retort was incorrect.  

In the end the new editor gave us his subjective opinion, based on the interpretations of an admittedly biased reviewer (he admitted he had reviewed our work at other journals) who withdrew from the review process instead of replying to our arguments. 

Not exactly the thorough, intellectual conversation that the peer-review process plays itself to be.

Had this been my first experience like this, I would let it go.  Disappointment comes with being a scientist.  However, it is sad to say that this isn't an unusual circumstance in modern scientific publishing these days.

So what can I do besides submitting to another journal?  Well I've taken the step of delivering an open letter to the editors at Frontiers in Human Neuroscience to outline my frustrations.  You can read it here or see it below.  

"Why make it an open letter?" you may ask.  I'm not doing it to be vindictive.  I'm doing it in the spirit of open access.  With increasing demand to open up access to articles, I believe it's imperative to also open up access to critiques and criticisms of the review process itself.   We don't have a formal repository of abuses of the peer-review process (i.e., a Yelp of scientific publishing to know editorial, reviewer, or journalistic biases before submitting manuscripts).  Often young scientists only find these out through trial-and-error when trying to publish their results.

Shedding light on the flaws of the peer-review process with your colleagues is the only way to start addressing these problems. Until we have a formal system for bringing these experiences to light, I'd like to start serving that role.   If you have an experience you'd like to share about your own experiences that highlight either errors/flaws or cases where it worked precisely how you think it should, please send them along.  I'll post your stories here on my blog as they come along and respect your privacy.

As of the time of this post, the Frontiers editors have not replied to my concerns laid out in the letter.  I'm not holding my breath.

Epilogue: For those of you interested in what a hostile review looks like, you can read the entire critique from the problematic reviewer and our reply to their critique here since Frontiers won't likely be releasing this anytime soon.

Open letter to the editors of Frontiers in Human Neuroscience:

Dr. HH,

Note: In the letter that follows I speak only for myself and not my fellow co-authors. I will be making this letter open to my scientific colleagues so they can understand what they can expect from Frontiers.

I am extremely displeased with the way in which Frontiers has handled the review of this manuscript. Not only has it been mishandled and delayed due to lack of foresight and administration by the editorial staff, but the reasoning behind the final decision rests on a tautological adherence to the reputation of a reviewer who withdrew from the process altogether without addressing the merits of our arguments. At no place in the decision were our responses acknowledged and shown to be incorrect.

My specific complaints include:

1) Our original editor, Dr. NC removed himself as editor in the middle of the review process instead of either seeing the submission through to completion or not accepting it if he knew he was stepping down.  Dr. NC ignored several email requests from our group asking about the status of the manuscript after we replied to the initial concerns.  We received no correspondence from Frontiers for more than two months and no reply from Dr. NC altogether.

2) It took the Frontiers office almost 8 weeks to revisit our manuscript after the original reviewers had either withdrawn (Reviewer 1) or signed off (Reviewer 2) on our changes. 

3) The editorial board chose to ignore the rude and unprofessional tone of Reviewer 1 (who even admitted that he has been a hostile reviewer of this manuscript at previous journals and yet did not decline the invitation to review).  This reviewer did not provide constructive critiques that could improve the quality of the study. Instead they made rude, accusatory and unprofessional statements that were provably false.  At no point did the editorial board intervene in this review. Further, the editorial decision completely ignored the second reviewer’s positive comments about the manuscript (some of which directly conflicted with the first reviewer).

4) Most insulting of all, rather than find an additional reviewer to replace the hostile referee who withdrew from the review, you made an executive decision based solely on the opinions of the withdrawn review! The arguments laid out in our reply were outright ignored, as was the fact that the reviewer had been withdrawn from the review process. If the reviewer had felt merit in their argument they would have further replied to our revisions.  Instead they chose to simply withdraw from the discussion without providing critical feedback.

5) The basis of your executive decision was based solely on several demonstrably false statements that we laid out in our response and we again describe here.  These include:

a) "One expert in the field of this paper had serious concerns about the design and interpretation of this study. The authors' responses did not alleviate the reviewer's concerns." 

Reply: This comment comes from a withdrawn review so it is unclear whether we actually alleviated the concerns of the reviewer.  However, if the reviewer contacted Frontiers after our reply and did not address our carefully laid out arguments, then this puts us in a grossly unfair position since there is no formal critique with which to make a response. 

b) "In particular, the reviewer was concerned that the 3 groups (normal, overweight, and obese) had major age differences, that these age differences may explain some of the results, and thought that the analysis attempting to factor age out is insufficient. The associate editor and I agree that considering the small number of subjects indeed the age differences cannot be overlooked." 

Reply: This argument is specious and wholly unsubstantiated on several levels.  First, as we carefully lay out in our reply, it is the Overweight group that is slightly older than the Normal and Obese groups.  In fact the ages of the Normal and Obese groups are statistically identical.  Yet our neural effects are Normal < Overweight < Obese in the categorical analysis (specifically Normal < Obese) and strictly linear in the parametric regression analysis.  If age was the driving factor our categorical effect pattern would be Normal < Obese < Overweight and have an inverted-U characteristic in the parametric analysis (Figure 7a).  Therefore this argument fails the simple logic test.

Second, the reviewer's belief that using nuisance regressors is an invalid way of controlling for non-specific effects goes against basic statistical theory.  The argument of accounted variance in nested regression models is the fundamental theory of structural equation modeling and mediation analyses.  Yet, the reviewer mis-quotes one article in a psychiatry journal to make their case.  We clearly demonstrated in the reply that BMI and age are not correlated and therefore (even according to the article cited by the withdrawn reviewer) age can be treated as a valid covariate for the neural effects.

Finally, an N=29 is NOT a small sample-size for conventional fMRI study.  There might be a case for it to under-power the categorical analysis, but in the parametric regression analyses (which supports the categorical findings) this is sufficient even for the conservative sample size recommended by Thirion et al. 2007 which the reviewer himself cites.  

At no stage has the withdrawn reviewer or editor pointed out why any of our arguments are incorrect.  They are simply disregarded without reply.

c) "Moreover, the mere fact that such conspicuous age difference were found, suggests some non-random sampling, which suggests that other factors would covary with BMI. In fact, overweight and especially morbid obesity are associated with numerous factors which may be of importance (e.g. socio-economic factors, health problems, not to speak of comfort in lying in the scanner), but the paper unfortunately does not provide clinical or demographic data on the subjects."

Reply: We fail to see the logic in the argument of "non-random sampling." In fact, with random sampling you will get some inconsistencies between groups.  Equal demographics across groups is only achieved through NON-RANDOM sampling.  Nonetheless, as we carefully laid out in our reply, there is no logical way that age can explain our effects.  As for the other covariates, the slippery slope argument to control for an ever increasing number of factors is a needless "torpedo" argument that can be lodged against virtually any between-group study.  This is a proof of concept study of inhibitory control deficits in obesity.  Once an effect is confirmed, further work can be done to elucidate mechanisms and causes.  But you have to know what you're looking for first.  If we followed your logic to its conclusion, no study could be published until all explanatory factors can be fully accounted for.  For example, differences between younger and older adults could be due to lack of comfort in the MRI environment in older adults.  Differences between borderline personality disorder and controls could be due to emotional reactions to the MRI environment.  Differences between schizophrenics and controls could be due to discomfort and irritation lying in a tight enclosed space.  In short, any between-subject study (including many of those published by the editors of this journal) could be said to have the same limitations as our study.  This in no way undermines studies of age, schizophrenia, borderline personality disorder, autism, or any other disease.  Instead, it emphasizes that despite these limitations in between-group comparisons there can be important information gained.  Similarly, our study on obesity is not immune to these issues, but it does not mean that important information on brain health in obesity could not be gleaned from this work.

d) "Another issue is the fact that since the behavioral results did not in fact confirm the prediction of the authors (more inhibitory problems with obesity), the paper relies heavily on reverse inference – that is, on drawing conclusions from assumptions on the cognitive roles of specific activated brain regions."

Reply: In many brain imaging studies, behavioral effects are weaker or not present where evoked brain dynamics are clearly visible.  In fact, the overall slowing of response times across groups is behavioral evidence for our argument that in higher BMI subjects, it requires more processing power to get the same behavioral output.  Thus all responses would be slower.  Again, this suggests that the manuscript was not clearly read since we make this point very clear in the text.

Given this rude and unprofessional experience, I will request that you no longer ask me to be a reviewer on future manuscripts since I do not wish to participate in such a flawed and obviously biased system.  I will also not be sending any new manuscripts to Frontiers in the foreseeable future. 

Sincerely,

Timothy Verstynen Ph.D.

Research Associate

Department of Psychology

Learning Research and Development Center

University of Pittsburgh

Pittsburgh, PA 15260