"Peer review is at the heart of scientific publishing. But its rigor has come into question lately—and news that a major publisher has retracted scientific papers could point to a wider peer review problem in progress.
Retraction Watch’s Cat Ferguson spoke with Elizabeth Moylan, BioMed Central’s senior editor of scientific integrity, who noted that some of the peer review concerns stem from suspicions about third-party involvement:
"Some of the manipulations appear to have been conducted by third-party agencies offering language-editing and submission assistance to authors. It is unclear whether the authors of the manuscripts involved were aware that the agencies were proposing fabricated reviewers on their behalf or whether authors proposed fabricated names directly themselves."
When Barbash spoke to BioMed Central’s associate editorial director for research integrity, she confirmed that the publisher had received a number of “very convincing” peer review reports.
However, odd email addresses and multiple reviews in different specialized fields raised red flags, which were confirmed when they learned that the scientists who had supposedly penned the reviews “hadn’t written them at all.”
But the problem doesn’t seem to be unique to BioMed Central. Barbash notes that the Committee on Publication Ethics issued a statement warning of “systematic, inappropriate attempts to manipulate the peer review process of several journals across publishers.” It’s an issue that has galvanized people like the anonymous founders of PubPeer, an anonymous feedback tool that lets scientists provide post-publication peer review without revealing their identities.
“While standard ‘pre-publication’ peer review often does improve the quality of published work, it is also clear that it lets through a huge number of mistakes, and a surprising amount of misconduct,” they told Vox’s Julia Belluz earlier this month.
“And the system as it stands has great difficulty in correcting work once published.”
For now, BioMed Central is retracting, rather than correcting, the falsely peer-reviewed articles."
=======================
More about PubPeer, link in above article. "Only two things can go wrong with a mathematics paper: it can contain a visible mistake, or it can be plagiarized." In life sciences, "there are many more opportunities for mistakes and dishonesty,"...
3/14/15, "Why you can't always believe what you read in scientific journals," Vox.com, Updated by Julia Belluz
"When people talk about the flaws in the scientific process, they often raise the problem of peer review.
Right now, when a researcher submits an article for publication in a journal, it's sent off to his or her peers for constructive criticism or even rejection. The idea is that science is like a self-cleansing machine: the scrutiny of one's colleagues is meant to ensure quality and keep the junk at bay.
But there are problems with this traditional "pre-publication" peer-review model: it relies on the goodwill of scientists, who are increasingly pressed and may not spend the time required to properly critique a work; it's subject to the biases of a select few; it's slow; and it sometimes fails. This means that even in the highest-quality journals, mistakes, flaws, and even fraudulent work make it on the record.
In the past, there was no way to publicly point out those errors, barring the rare event of a retraction.
Now, the (anonymous) founders of a new website, PubPeer, are trying to change that. Since establishing the site in 2012, they've grown into a global platform for "post-publication" peer review — a new method some say could replace the current model.
PubPeer researchers can now comment on scientific
articles, critiquing and discussing works anonymously, as soon as
they've been published in journals — like a comments section on a news
site. The space for criticism is no longer confined to just a few
reviewers; anyone can log in and leave a thoughtful remark.
PubPeer has already amassed more than 25,000 comments in its centralized database. It has helped uncover science fraud, and recently became the subject of a court case over the right to anonymous scientific discussion. Along the way, it's creating a successful model that could replace the hallowed — and flawed — traditional peer-review process. On the condition of anonymity, I interviewed the PubPeer founders — an early career researcher, computer scientist, and graduate student — about the revolution in peer review they're trying to shape, why science commenters tend not to troll, and why they refuse to reveal their identities.
PubPeer has already amassed more than 25,000 comments in its centralized database. It has helped uncover science fraud, and recently became the subject of a court case over the right to anonymous scientific discussion. Along the way, it's creating a successful model that could replace the hallowed — and flawed — traditional peer-review process. On the condition of anonymity, I interviewed the PubPeer founders — an early career researcher, computer scientist, and graduate student — about the revolution in peer review they're trying to shape, why science commenters tend not to troll, and why they refuse to reveal their identities.
PubPeer Organizers: While standard "pre-publication" peer review often does improve the quality of published work, it is also clear that it lets through a huge number of mistakes and over-interpretations, and a surprising amount of misconduct. And the system as it stands has great difficulty in correcting work once published.
JB: How does PubPeer address these issues?
PP: PubPeer enables centralization of all commentary about published work. Instead of two or three pre-publication referees producing a confidential report to a strict deadline, any expert in the world can add to information about a paper after careful study, and that information is available to all.
A distinguishing feature of PubPeer is that we allow anonymous comments, including an option for strong, user-controlled anonymity. This has some potential disadvantages (which experience suggests are often over-dramatized) but has clearly enabled many serious problems to be brought to light.
JB: Can you talk a bit about how you created PubPeer? What sparked the idea?
PP: The idea for PubPeer was born of the frustration of observing so many uncorrected and unacknowledged flaws in published work — a very widespread sentiment among researchers. We know from discussions with colleagues that the idea of a commenting site was not necessarily original. Several precursor sites were launched. But it seems we were the first to find a formula and a platform that prospered.
JB: How many people are using the site? And where are they coming from?
PP: The top countries connecting to the site are the United States, the United Kingdom, Germany, Japan, France, Canada, Israel, Brazil, Switzerland, and Italy. We assume they are all scientists.
JB: In journalism, a fair amount of trolling happens in comments sections. How do you avoid that? What kind of vetting of the comments goes on?
PP: The key point non-scientists struggle to understand is that effective criticism of science must have genuine substance. It's extremely difficult to invent a damaging criticism of a good paper, even if you want to. Just saying something is crap or nitpicking is deeply unconvincing. We normally moderate such comments, but our users are also quite effective at policing these issues. Our commenting and moderation policies reinforce this by requiring comments to be based on publicly verifiable information. So the whole issue of conflicts of interest and trolling is not nearly as serious as elsewhere: a comment has force if it makes a good point that a reader can check, and if it does, it matters little who made it.
JB: Have you noticed any trends in who comments or in which disciplines? What types/fields of scientists are the most active users?
PP: The bulk of our commenting is in the life sciences, but PubPeer is open for discussion of all publications. Nanotechnology, chemistry, climate science, management studies, and quantum mechanics all have active communities on PubPeer.
JB: Why do you think you have so many life sciences commenters?
PP: An interesting question is the degree to which different fields are subject to misconduct. Thus, it seems only two things can go wrong with a mathematics paper: it can contain a visible mistake, or it can be plagiarized. By contrast, in much of biology, readers have to trust the authors to describe their experiments accurately, and there are many more opportunities for mistakes and dishonesty. In addition, it seems particularly easy to spot problems in image data, which abound in life sciences. A recent study reported on PubPeer found that fully 25 percent of a random sample of cancer research publications contained problems in image data. That is a shocking statistic.
JB: What impact have you had so far? Has PubPeer resulted in any major findings of fraud or retractions or the like?
PP: There are now more than 25,000 comments in the PubPeer database, which represent a great deal of collected wisdom and expert information available to other scientists.
In terms of helping uncover fraud, PubPeer channeled many of the questions about the STAP stem-cell fiasco last year [when a famous stem cell paper came under fire after anonymous commenters on PubPeer pointed out big flaws in the research]. We are aware of several cases where researchers appear to have lost jobs after incisive commentary appeared on PubPeer, and there are certainly many high-profile researchers whose work has been called into question on the site.
JB: There's been a lot of talk in recent years about how broken science is, particularly the peer-review process. What are the bigger systemic changes that need to happen in order to fix it?
PP: The biggest problem is the pressure to chase after "metrics" — indirect measures of scientific success. The most important metric is publication in top journals, which determines jobs, grants, everything. This distorts the scientific process toward mostly illusory "breakthroughs" and "high-impact research" at the expense of careful work. Scientists now find themselves ruled by often-incompetent kingmakers — the editors of the top journals — who effectively decide their futures and make scientific fashion.
PubPeer is helping scientists retake control of their lives, work, and careers by providing a collective judgment that is independent of and ultimately more important than acceptance by the top journals. That judgment is the expert opinion of your peers. We are also big fans of open-access publishing and the use of pre-print servers such as ArXiv or the newer bioRxiv; we believe these will also loosen the stranglehold of the top journals on research.
JB: Finally: why do you refuse to reveal your identities? You're critiquing science in an open democracy, not offering political dissent in a repressive regime or something.
PP: You need look no further than the Sarkar suit for an answer. [Fazul Sarkar is a cancer researcher who sued PubPeer, demanding the identities of anonymous commenters who criticized his work; the court sided with PubPeer.] There is a good chance that we as organizers would have been sued by now or suffered some kind of reprisal — rejected papers, non-funded grants, etc. And certainly we would have come under pressure to remove posts about colleagues or reveal the identity of commenters. That said, we do not expect or plan to remain anonymous indefinitely, but it has helped protect us while we build up the platform."
=======================
--------------------------------------
More on "famous stem cell paper" mentioned in above Vox article: Japanese author is disgraced, American co-author skates:
"In the US, investigations into scientific misconduct usually take place under a veil of secrecy. In all likelihood, Brigham has begun its own inquiry [into Vacanti's involvement] but, in stark contrast to the one carried out by Riken [in Japan], we probably won’t learn anything about it – even the fact of its existence – until after a verdict is reached." Vacanti himself is on a year’s sabbatical from Brigham."...
.
2/18/15, "What pushes scientists to lie? The disturbing but familiar story of Haruko Obokata," UK Guardian, John Rasko and Carl Power
.
"And what of Obokata’s US colleagues? More particularly, what of Charles Vacanti, the chief co-author of the now-discredited Stap cell papers?
 |
| Charles Vacanti, 3rd from left |
Unfortunately, that humble boast backfired. No one else could get his recipe to work. Many hoped that Vacanti would toil night and day until he proved the existence of Stap cells, but instead, last September, he left Brigham for a year’s sabbatical. Had he lost faith in Stap cells? Apparently not. As a parting gesture, Vacanti posted an improved recipe which, he said, “should increase the likelihood of success”. So far, the new recipe seems no better than the old one.
Vacanti’s role in this scientific debacle is especially intriguing because Stap cells originally sprang from his fertile imagination. For well over a decade, he had been working on a hunch that pluripotent stem cells exist in all mammalian tissue, ready to swing into action whenever needed. It was a big, bright, potentially career-defining idea which for a long time Vacanti couldn’t sell. He lacked conclusive proof. He also lacked credibility. After all, he was not a stem cell scientist but an anesthesiologist and tissue engineer best known for grafting an artificial ear on to the back of a mouse (the infamous Vacanti earmouse)....
The year 2014 was one of extremes for Haruko Obokata....Barely 30 years old, she was head of her own laboratory at the Riken Center for Developmental Biology (CDB) in Kobe, Japan, and was taking the male-dominated world of stem cell research by storm. She was hailed as a bright new star in the scientific firmament and a national hero. But her glory was short-lived and her fall from grace spectacular, completed in several humiliating stages.
Obokata shot to prominence in January 2014 when she published two breakthrough articles in Nature, one of the world’s top science journals. She and her colleagues had demonstrated a surprisingly simple way of turning ordinary body cells – she used mouse blood cells – into something very much like embryonic stem cells."...
.
No comments:
Post a Comment