Saturday, December 10, 2011

Tales From The Science Trenches: The problem of journal addiction

Note: Over the next few months I'll be doing a series of posts critiquing modern peer-review process.  As part of this I'd like to collect stories from other scientists on their experiences trying to get papers through peer-review.  Good or bad, if you have a story you'd like to share, please email me (timothyv[at]gmail[dot]com) or put them in the comments section and I'll post them as I see them.  

Today's guest post comes from Joern Diedrichsen who is one of the smartest, most creative scientists that I know.  He studies motor control at University College London and has a provocative take on the pains of the current peer-review process and the addictive mentality that keeps bringing us back for more.

The problem of journal addiction

"Let’s admit it, we are all suckers for glossy journals. Nature, Science, Neuron, Nature Neuroscience... oh, how we puff up our chest when we have a paper accepted in a high place! How we strut around and announce in talks: “our new paper in xxx shows…”. And how deflated, angry, and bitter are we after a rejection.  How we hate the reviewers and the editors. "Ignorant bastards just do not understand anything."

And of course, we are correct. Our papers are misinterpreted, rejected for selfish reasons, or because of plain ignorance. And the editors do not have the spine to stand up and tell reviewers how petty they are.  And the next time WE get asked to review a paper for this journal – which is really not as good as our rejected paper – we’ll show them! The review request lands in our inbox and we metamorphose into the dreaded reviewer 2. Or 3 (depending on how many hours past lunch it is).

Of course we could boycott the whole system. Retreat to a small island. Only send papers (no matter how good) to PLOSone or Frontiers. But a month later, when the wounds have healed, we find ourselves preparing a cover letter for another submission to one of the hated journals. Talk about addiction.

Do we really think that the title of the journal we publish in means this much? Considering the degree to which professional editors are slaves to fashion, and how random the review process is, we really shouldn’t. I think some of my weaker papers have been published in “better” journals - and vice versa. There is a slight positive correlation – but not very high.  Many of the papers that in retrospect are important, get cited, and have impact on the field are in 2nd-tier journals. But then again, in terms of careers, candidate selection, and funding decisions, we all like to rely on the fast heuristic of the impact factor, not on how important we think the paper is.

So, rather than go by the journal name, if we all would READ the actual papers, see how clear, compelling and novel the results really are, we should be able to break the yoke that editors and reviewers hold over us, right? So, why are we not online every week making sure that good papers in 2nd tier journals get their well-earned exposure by posting online evaluations on journal websites? Why do the online debates in PLOS Comp Biology often only consist of statements such as “The reference section is missing a crucial reference: My article, 2011”? Why does Faculty of 1000 seem to be struggling in terms of relevance and in getting enough submissions? I guess we are simply too busy writing angry reviews, rewriting our own papers for the next glossy journal, or arguing with journal editors. And trust me, life as an editor is not rosy either. Talk about a thankless job.

Currently, I do not see a good way out. Do we really think that low-threshold mass-journals like Frontiers and PLOSone are the solution? It seems there is just too much stuff out there, and post-hoc online evaluations by people in the field seem not to work very well. So maybe the traditional peer-review and tiered journal system is – like western democracy and capitalism – the lesser of the evils…. But maybe we can start to not reject because we feel the paper is too novel, doesn’t cite us enough, or infringes on our turf? Maybe we should stop pushing papers by friends in high profile journals for political gain? Maybe we should stop evaluating people based on where they publish and turn our attention to the science they produce instead? Perhaps teach our students some integrity and honesty?

No time for that…. I need to fine-tune that cover letter…"

Monday, December 5, 2011

Tales From The Science Trenches: Case of the missing editor

Note: Over the next few months I'll be doing a series of posts critiquing modern peer-review process.  As part of this I'd like to collect stories from other scientists on their experiences trying to get papers through peer-review.  Good or bad, if you have a story you'd like to share, please email me (timothyv[at]gmail[dot]com) or put them in the comments section and I'll post them as I see them.  

Almost immediately after my post yesterday a friend emailed me to share her story.  I think it sets another good case for why proper editorial management is so important.

If you or a colleague have a similar story or even a positive one about the submission process, please email me your tale.

Case of the Missing Editor

"Hi Tim,

I submitted a manuscript to the International Journal of [redacted] on March 7.  This journal doesn’t have an online submission site, so authors are supposed to email (or paper mail) the submission to the editor.  After several attempts to email the editor with the email address listed on the journal website and on his university web page, none of the emails went through. I also didn’t get a response to an email I sent to someone else on the editorial board looking for the editor’s email address.

So I contacted someone in the office I work in to see if they had ideas about where to go from there.  It turned out the email addresses listed for the editor were incorrect, so they gave me the right one.  I sent in the manuscript, it seemed to go through, but I never received acknowledgement of receipt.  I forgot about it for a while, and figured I’d hear from them with a decision in a few months.

At the end of July, I sent an email to the editor asking for acknowledgement of receipt and information on when I might expect to hear a decision.  He never responded.  At the end of August, I emailed an associate editor at the journal who happened to be at the same university, and asked if he could help me out with getting in contact with the editor about my questions.  He responded right away saying he would get in contact with the editor for me, but I never heard anything from him after that.

At the beginning of September, my advisor emailed the editor asking about my manuscript and another of his that he had sent in May.  The editor responded within a day, saying he received the manuscripts and sent them out for review, and he has a new email address that we should use.

We heard nothing back by the end of October, so at that point I sent another email inquiring about both manuscripts that are under review.  Three weeks later, he responded and apologized for the 'long delay' in his response, saying that he has had a 'high workload.'  He said 'I will be able to inform you about a decision very soon now. You will hear from me before December 15.'

And so we wait.

I’ve considered (many times) pulling my manuscript from the journal and submitting it somewhere else, but haven’t because the topic of my paper just wouldn’t fit well at very many journals, and all of my other journal options have lengthy (2-3 year) waits between acceptance and publication.  Needless to say, I don’t plan on ever submitting to this journal again, at least as long as this person remains the editor."

Sunday, December 4, 2011

Tales From The Science Trenches: An open-letter to Frontiers in Human Neuroscience

Note: Over the next few months I'll be doing a series of posts critiquing modern peer-review process.  As part of this I'd like to collect stories from other scientists on their experiences trying to get papers through peer-review.  Good or bad, if you have a story you'd like to share, please email me (timothyv[at]gmail[dot]com) or put them in the comments section and I'll post them as I see them.  

In an ideal sense, the peer-review process is designed to provide useful critiques of scientific manuscripts so that the end result has minimal errors and justified conclusions.

Unfortunately, personal biases, poor editorial oversight and general abuse of the review process can sometimes impair communication of scientific results.  What is described below is one example of the flaws in the process that are sometimes experienced by those trying to publish scientific results.

Recently my colleagues and I tried submitting a manuscript to the journal Frontiers in Human Neuroscience.  Frontier is an open access set of journals that prides itself for a rapid turn-around (i.e., fast review cycles) and a simplified peer-review process focused primarily on methodological validity.  

The manuscript we submitted was a study on the brain activation changes that occur with increasing obesity.   The paper had already been bounced around to several other journals before being sent to Frontiers.  As sometimes happens, the same anonymous reviewer followed us from journal to journal providing the same general (and in our opinion incorrect) critiques of the manuscript.

Now for those of you outside the process, it's generally considered bad manners to agree to review a manuscript that you have already critiqued at another journal.  The logic being that you can't provide an unbiased review of the manuscript because you've already judged it previously.

Unfortunately, this particular reviewer keeps agreeing to critique the manuscript while letting his previous judgements color his critique of our manuscript.  What's worse is that he (I'm assuming the gender because I have a good idea who it is) isn't interested in making helpful criticisms that could expand or fix the paper.  You can see for yourself in the linked documents below.

So we submitted our work to Frontiers and this reviewer followed us there.  He gave his usual critique, ignoring many of the changes we that made to address these concerns.  We then spent a couple of months further revising the manuscript and running control analyses.   Once the revisions were submitted to the journal, this hostile referee immediately withdrew his review (again, rather than address our arguments) while the only other referee signed off on our changes as being sufficient for publication.

Then we waited 8 weeks with no word from Frontiers.  Again, for those of you non-scientists this is a pretty long time to wait for an editorial decision after the reviewers had submitted their responses.  Emails to the editor were ignored and the status remained unchanged.

It turns out that the original editor had retired after we resubmitted our manuscript.  Instead of seeing the submission through to completion (or not agreeing to be an editor in the first place), he simply withdrew as well.  It appears that it took Frontiers almost to figure this out.  Once a new editor was assigned, this individual (Dr. HH in the open letter below) made an executive decision to reject the manuscript based only on the withdrawn review.  He completely ignored our replies and made no mention as to why our carefully constructed retort was incorrect.  

In the end the new editor gave us his subjective opinion, based on the interpretations of an admittedly biased reviewer (he admitted he had reviewed our work at other journals) who withdrew from the review process instead of replying to our arguments. 

Not exactly the thorough, intellectual conversation that the peer-review process plays itself to be.

Had this been my first experience like this, I would let it go.  Disappointment comes with being a scientist.  However, it is sad to say that this isn't an unusual circumstance in modern scientific publishing these days.

So what can I do besides submitting to another journal?  Well I've taken the step of delivering an open letter to the editors at Frontiers in Human Neuroscience to outline my frustrations.  You can read it here or see it below.  

"Why make it an open letter?" you may ask.  I'm not doing it to be vindictive.  I'm doing it in the spirit of open access.  With increasing demand to open up access to articles, I believe it's imperative to also open up access to critiques and criticisms of the review process itself.   We don't have a formal repository of abuses of the peer-review process (i.e., a Yelp of scientific publishing to know editorial, reviewer, or journalistic biases before submitting manuscripts).  Often young scientists only find these out through trial-and-error when trying to publish their results.

Shedding light on the flaws of the peer-review process with your colleagues is the only way to start addressing these problems. Until we have a formal system for bringing these experiences to light, I'd like to start serving that role.   If you have an experience you'd like to share about your own experiences that highlight either errors/flaws or cases where it worked precisely how you think it should, please send them along.  I'll post your stories here on my blog as they come along and respect your privacy.

As of the time of this post, the Frontiers editors have not replied to my concerns laid out in the letter.  I'm not holding my breath.

Epilogue: For those of you interested in what a hostile review looks like, you can read the entire critique from the problematic reviewer and our reply to their critique here since Frontiers won't likely be releasing this anytime soon.

Open letter to the editors of Frontiers in Human Neuroscience:

Dr. HH,

Note: In the letter that follows I speak only for myself and not my fellow co-authors. I will be making this letter open to my scientific colleagues so they can understand what they can expect from Frontiers.

I am extremely displeased with the way in which Frontiers has handled the review of this manuscript. Not only has it been mishandled and delayed due to lack of foresight and administration by the editorial staff, but the reasoning behind the final decision rests on a tautological adherence to the reputation of a reviewer who withdrew from the process altogether without addressing the merits of our arguments. At no place in the decision were our responses acknowledged and shown to be incorrect.

My specific complaints include:

1) Our original editor, Dr. NC removed himself as editor in the middle of the review process instead of either seeing the submission through to completion or not accepting it if he knew he was stepping down.  Dr. NC ignored several email requests from our group asking about the status of the manuscript after we replied to the initial concerns.  We received no correspondence from Frontiers for more than two months and no reply from Dr. NC altogether.

2) It took the Frontiers office almost 8 weeks to revisit our manuscript after the original reviewers had either withdrawn (Reviewer 1) or signed off (Reviewer 2) on our changes. 

3) The editorial board chose to ignore the rude and unprofessional tone of Reviewer 1 (who even admitted that he has been a hostile reviewer of this manuscript at previous journals and yet did not decline the invitation to review).  This reviewer did not provide constructive critiques that could improve the quality of the study. Instead they made rude, accusatory and unprofessional statements that were provably false.  At no point did the editorial board intervene in this review. Further, the editorial decision completely ignored the second reviewer’s positive comments about the manuscript (some of which directly conflicted with the first reviewer).

4) Most insulting of all, rather than find an additional reviewer to replace the hostile referee who withdrew from the review, you made an executive decision based solely on the opinions of the withdrawn review! The arguments laid out in our reply were outright ignored, as was the fact that the reviewer had been withdrawn from the review process. If the reviewer had felt merit in their argument they would have further replied to our revisions.  Instead they chose to simply withdraw from the discussion without providing critical feedback.

5) The basis of your executive decision was based solely on several demonstrably false statements that we laid out in our response and we again describe here.  These include:

a) "One expert in the field of this paper had serious concerns about the design and interpretation of this study. The authors' responses did not alleviate the reviewer's concerns." 

Reply: This comment comes from a withdrawn review so it is unclear whether we actually alleviated the concerns of the reviewer.  However, if the reviewer contacted Frontiers after our reply and did not address our carefully laid out arguments, then this puts us in a grossly unfair position since there is no formal critique with which to make a response. 

b) "In particular, the reviewer was concerned that the 3 groups (normal, overweight, and obese) had major age differences, that these age differences may explain some of the results, and thought that the analysis attempting to factor age out is insufficient. The associate editor and I agree that considering the small number of subjects indeed the age differences cannot be overlooked." 

Reply: This argument is specious and wholly unsubstantiated on several levels.  First, as we carefully lay out in our reply, it is the Overweight group that is slightly older than the Normal and Obese groups.  In fact the ages of the Normal and Obese groups are statistically identical.  Yet our neural effects are Normal < Overweight < Obese in the categorical analysis (specifically Normal < Obese) and strictly linear in the parametric regression analysis.  If age was the driving factor our categorical effect pattern would be Normal < Obese < Overweight and have an inverted-U characteristic in the parametric analysis (Figure 7a).  Therefore this argument fails the simple logic test.

Second, the reviewer's belief that using nuisance regressors is an invalid way of controlling for non-specific effects goes against basic statistical theory.  The argument of accounted variance in nested regression models is the fundamental theory of structural equation modeling and mediation analyses.  Yet, the reviewer mis-quotes one article in a psychiatry journal to make their case.  We clearly demonstrated in the reply that BMI and age are not correlated and therefore (even according to the article cited by the withdrawn reviewer) age can be treated as a valid covariate for the neural effects.

Finally, an N=29 is NOT a small sample-size for conventional fMRI study.  There might be a case for it to under-power the categorical analysis, but in the parametric regression analyses (which supports the categorical findings) this is sufficient even for the conservative sample size recommended by Thirion et al. 2007 which the reviewer himself cites.  

At no stage has the withdrawn reviewer or editor pointed out why any of our arguments are incorrect.  They are simply disregarded without reply.

c) "Moreover, the mere fact that such conspicuous age difference were found, suggests some non-random sampling, which suggests that other factors would covary with BMI. In fact, overweight and especially morbid obesity are associated with numerous factors which may be of importance (e.g. socio-economic factors, health problems, not to speak of comfort in lying in the scanner), but the paper unfortunately does not provide clinical or demographic data on the subjects."

Reply: We fail to see the logic in the argument of "non-random sampling." In fact, with random sampling you will get some inconsistencies between groups.  Equal demographics across groups is only achieved through NON-RANDOM sampling.  Nonetheless, as we carefully laid out in our reply, there is no logical way that age can explain our effects.  As for the other covariates, the slippery slope argument to control for an ever increasing number of factors is a needless "torpedo" argument that can be lodged against virtually any between-group study.  This is a proof of concept study of inhibitory control deficits in obesity.  Once an effect is confirmed, further work can be done to elucidate mechanisms and causes.  But you have to know what you're looking for first.  If we followed your logic to its conclusion, no study could be published until all explanatory factors can be fully accounted for.  For example, differences between younger and older adults could be due to lack of comfort in the MRI environment in older adults.  Differences between borderline personality disorder and controls could be due to emotional reactions to the MRI environment.  Differences between schizophrenics and controls could be due to discomfort and irritation lying in a tight enclosed space.  In short, any between-subject study (including many of those published by the editors of this journal) could be said to have the same limitations as our study.  This in no way undermines studies of age, schizophrenia, borderline personality disorder, autism, or any other disease.  Instead, it emphasizes that despite these limitations in between-group comparisons there can be important information gained.  Similarly, our study on obesity is not immune to these issues, but it does not mean that important information on brain health in obesity could not be gleaned from this work.

d) "Another issue is the fact that since the behavioral results did not in fact confirm the prediction of the authors (more inhibitory problems with obesity), the paper relies heavily on reverse inference – that is, on drawing conclusions from assumptions on the cognitive roles of specific activated brain regions."

Reply: In many brain imaging studies, behavioral effects are weaker or not present where evoked brain dynamics are clearly visible.  In fact, the overall slowing of response times across groups is behavioral evidence for our argument that in higher BMI subjects, it requires more processing power to get the same behavioral output.  Thus all responses would be slower.  Again, this suggests that the manuscript was not clearly read since we make this point very clear in the text.

Given this rude and unprofessional experience, I will request that you no longer ask me to be a reviewer on future manuscripts since I do not wish to participate in such a flawed and obviously biased system.  I will also not be sending any new manuscripts to Frontiers in the foreseeable future. 

Timothy Verstynen Ph.D.
Research Associate
Department of Psychology
Learning Research and Development Center
University of Pittsburgh
Pittsburgh, PA 15260