Report Suggests That Only 1 in 10 Treatments Are Backed By Quality Evidence

The study led by Jeremey Howick who is the director of the Oxford Empathy Programme examined 154 systematic treatment reviews that were published between 2015-2019 finding that only 15 (9.9%) had high-quality evidence according to the gold standard method GRADE; 37% had moderate evidence, 31% had low, and 22% had very low-quality evidence to back their suggested course of action. 

These alarming findings highlight a decades-old concern about medical academia trending towards lower quality research which may propagate the undermining of future research based upon poorly supported evidence from previous trials that don’t stand up to proper standards of scrutiny. 

Although some in the scientific community argue that the gold standard is too strict to be useful in the first place, the researchers concede that it is possible that in limited circumstances the new study may be a reflection of the dilution of the research pool through the publication of too many poor-quality trials. 

PubMed for example now publishes at a rate of around 30 new trials per day or about 12,000 per year, and there are too many systematic reviews to synthesize all of these trials with 2,000 being published annually in PubMed alone. This once more is simply a matter of too much information with too little emphasis being placed on quality control. 

Denouncing the publish or perish mentality that appears to be pervasive in medical academic literature the researchers highlight the importance and urgent need for the quality of research to improve rather than the quantity, arguing that there is currently no evidence that the quality of research has actually improved over the last three decades.

Abstract:

Objectives

The objective of the study was to determine the change in quality of evidence in updates of Cochrane reviews that were initially published between January 1, 2013 and June 30, 2014. We used the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system to document evidence quality.

Study Design and Setting

We searched the Cochrane Database of Systematic Reviews on March 20, 2020 to identify which of the reviews from the initial (2013/14) sample had been updated. Using the same methods to determine the quality of evidence in the previous analysis, we assessed the quality of evidence for the first-listed primary outcomes in the updated reviews.

Results

Of the 608 reviews in the original sample, 154 had been updated with and 151 contained available data for both original and updated systematic reviews (24.8%). The updated reviews included: 15 (9.9%) with high-quality evidence, 56 (37.1%) with moderate-quality evidence, 47 (31.1%) with low-quality evidence, and 33 (21.9%) with very low-quality evidence. No change in the GRADE quality of evidence was found for most (103, 68.2%) of the updated reviews. The quality of evidence rating was downgraded in 28 reviews (58.3%) and upgraded in 20 (41.7%), although only six reviews were promoted to high quality.

Conclusion

Updated systematic reviews continued to suggest that only a minority of outcomes for health care interventions are supported by high-quality evidence. The quality of the evidence did not consistently improve or worsen in updated reviews.