Tuesday, May 26, 2015

Bad incentives in peer-reviewed science

The inability of the peer-review process to detect fraud and error in scientific publications is getting some mainstream attention. Adam Marcus and Ivan Oransky, the founders of Retraction Watch, had an op-ed in the New York Times entitled What's Behind Big Science Frauds?, in which they neatly summed up the situation:
Economists like to say there are no bad people, just bad incentives. The incentives to publish today are corrupting the scientific literature and the media that covers it. Until those incentives change, we’ll all get fooled again.
Earlier this year I saw Tom Stoppard's play The Hard Problem at the Royal National Theatre, which deals with the same issue. The tragedy is driven by the characters being entranced by the prospect of publishing an attention-grabbing result. Below the fold, more on the problem of bad incentives in science.

Back in April, after a Wellcome Trust symposium on the reproducibility and reliability of biomedical science, Richard Horton, editor of The Lancet, wrote an editorial entitled What is medicine’s 5 sigma? that is well worth a read. His focus is also on incentives for scientists:
In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory. Or they retrofit hypotheses to fit their data.
and journal editors:
Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals. Our love of "significance" pollutes the literature with many a statistical fairy-tale. We reject important confirmations.
and Universities:
in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices.
Horton points out that:
Part of the problem is that no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative.
He concludes:
The good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system.
Six years ago Marcia Angell, the long-time editor of a competitor to The Lancet wrote in an review of three books pointing out the corrupt incentives that drug companies provide researchers and Universities:
It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.
In most fields, little has changed since then. Horton points to an exception:
Following several high-profile errors, the particle physics community now invests great effort into intensive checking and re-checking of data prior to publication. By filtering results through independent working groups, physicists are encouraged to criticise. Good criticism is rewarded. The goal is a reliable result, and the incentives for scientists are aligned around this goal.
Unfortunately, particle physics is an exception. The cost of finding the Higgs Boson was around $13.25B, but no-one stood to make a profit from it. A single particle physics paper can have over 5,000 authors. The resources needed for "intensive checking and re-checking of data prior to publication" are trivial by comparison. In other fields, the incentives for all actors are against devoting resources which would represent a significant part of the total for the research to such checking.

Fixing these problems of science is a collective action problem; it requires all actors to take actions that are against their immediate interests roughly simultaneously. So nothing happens, and the long-term result is, as Arthur Caplan (of the Division of Medical Ethics at NYU's Langone Medical Center) pointed out, a total loss of science's credibility:
The time for a serious, sustained international effort to halt publication pollution is now. Otherwise scientists and physicians will not have to argue about any issue—no one will believe them anyway.
(see also John Michael Greer). I am not optimistic, based on the fact that the problem has been obvious for many years, and that this is but one aspect of society's inability to deal with long-term problems.

10 comments:

David. said...

Tne New York Times editorial board today weighs in with Scientists Who Cheat. It fails to understand the roots of the problem, pointed out in the earlier op-ed by Marcus and Oransky, and ends, ironically, with a call for more research.

David. said...

Lambert Strether at Naked Capitalism posts a piece by Roy Poses MD that contrasts the New England Journal of Medicine's editorial stance that concerns about Big Pharma's money influencing health care are overblown with the British Medical Journal's response from two former NEJM editors and a former NEJM national correspondent. Poses illustrates this debate by the example of the recent American Society of Clinical Oncology (ASCO) meeting at which it was noteworthy that a single speech mentioned that cancer drugs were becoming unaffordable:

"It is unprecedented for plenary speeches, which typically address scientific and medical issues, to substantially take on the topic of drug costs, said Alan Venook, a professor of medicine ... The prominent venue for the speech was also unusual because, like many medical meetings, ASCO is sponsored by pharmaceutical companies and often focuses on highlighting advancements in drug development, said Dr. Venook. He said discussing drug prices there is ‘uncomfortable’ because it could be seen as ‘biting the hand that feeds you.’ ... Doctors are also reluctant to antagonize the drug industry because they need pharmaceutical firms to invest in developing new medicines for patients, he said. ... ‘It’s a tough balancing act for ASCO where the meeting is largely funded by pharma,’ Dr. Venook said in an interview. ‘You can’t have a [plenary] talk trashing pharma, but you can have a talk by a respected person questioning it.‘"

No kidding. As Poses reports:

"ASCO ... only gets 16.1% of its revenue from member dues. Thus a ostensible membership organization gets only about a sixth of its funding from members’ dues. ... Yet the organization has become quite wealthy. Its most recent (2013) US Internal Revenue Service 990 Form reveals that it owns over $55 million in real estate, and has over $104 million in investments (presumably as an endowment.) The organizations’ leaders are also doing very well. Its CEO, Allen Lichter MD, got $804,775 in total compensation in 2012. ... The source of all that money seems mainly to be pharmaceutical and other health care corporations ... the ASCO annual report does list 23 pharmaceutical and biotechnology companies, and one for-profit cancer hospital chain as contributing at least $1 million each in total to the non-profit over time. The report lists 37 pharmaceutical, biotechnology, and medical device companies as current corporate donors, and also 10 other for-profit health care related corporations as current corporate donors."

David. said...

Ars Technica reports on three articles in the current issue of AAAS Science that set out proposals for changing the incentives for scientific publication in ways that would reduce fraud and abuse.

David. said...

Australian academics are up in arms:

"Emeritus professor Stephen Leeder was sacked by the Medical Journal of Australia (MJA) in April after challenging a decision to outsource some of the journal’s functions to the world’s biggest scientific publisher, Elsevier."

The state library of New South Wales is hosting a symposium on the rapacious economics of scholarly publishing. Leeder says:

“Scholarly publishing is a bit like the Trans-Pacific Partnership Agreement. It’s not totally clear what the hell is going on, but you can be sure someone is making a hell of a lot of money from it.”

David. said...

Mandatory pre-registration of clinical trials seems to have revealed endemic suppression of null or negative results. Robert Kaplan and Veronica Ivin report:

"17 of 30 studies (57%) published prior to 2000 showed a significant benefit of intervention on the primary outcome in comparison to only 2 among the 25 (8%) trials published after 2000 (χ2=12.2,df= 1, p=0.0005). There has been no change in the proportion of trials that compared treatment to placebo versus active comparator. Industry co-sponsorship was unrelated to the probability of reporting a significant benefit. Pre-registration in clinical trials.gov was strongly associated with the trend toward null findings."

David. said...

The New York Times reports on a paper in AAAS Science that describes an effort to reproduce 100 results from the top 3 psychology journals.

"Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance — 35 of the studies held up, and 62 did not. (Three were excluded because their significance was not clear.) The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. Yet very few of the redone studies contradicted the original ones; their results were simply weaker."

David. said...

Der Spiegel interviews British cardiologist Peter Wilmshurst: Whistleblower on Medical Research Fraud: 'Positive Results Are Better for Your Career'. Well worth a read.

David. said...

Andrew C. Chang and Phillip Li of the Federal Reserve tried to replicate the results in 67 papers from 13 top economics journals. In most cases they had access to the code and data and help from the authors. They replicated 22/67 without help from the authors, and 22/59 with help. That's less than half.

David. said...

NEJM weighs in, getting the point that it is bad incentives that cause people to subvert the peer review system.

David. said...

Functional neuroimaging shows that scientists have been conditioned to journal impact factor in making publication decisions despite its total bogosity.