Afroscientific

The Positive Results Bias in Science

The website catalogofbias.org summarises positive results bias as "the tendency to submit, accept and publish positive results rather than non-significant or negative results."

It continues: “positive results bias occurs because a considerable amount of research evidence goes unpublished, which contains more negative or null results than positive ones. This leads to spurious claims and overestimation of the results of systematic reviews and can also be considered unethical. Non-publication of results can also lead to research wastage as researchers may unnecessarily repeat studies because the results are unpublished.”

An August 2018 study in Psychological Medicine demonstrated the positive results bias in trials on antidepressant drugs. In “The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression”, the researchers looked at 105 drug trials that had been registered with the US Food and Drug Administration. They explain that “pharmaceutical companies must preregister all trials they intend to use to obtain FDA approval; hence, trials with non-significant results, even if unpublished, are still accessible.”

This is an important point which we’ll return to. But the main findings of the study make for startling reading:

Of 105 antidepressant trials, 53 (50%) trials were considered positive by the FDA and 52 (50%) were considered negative or questionable (Fig. 1a). While all but one of the positive trials (98%) were published, only 25 (48%) of the negative trials were published. Hence, 77 trials were published, of which 25 (32%) were negative (Fig. 1b). Ten negative trials, however, became ‘positive’ in the published literature, by omitting unfavorable outcomes or switching the status of the primary and secondary outcomes (Fig. 1c). Without access to the FDA reviews, it would not have been possible to conclude that these trials, when analyzed according to protocol, were not positive. Among the remaining 15 (19%) negative trials, five were published with spin in the abstract (i.e. concluding that the treatment was effective). For instance, one article reported non-significant results for the primary outcome (p = 0.10), yet concluded that the trial ‘demonstrates an antidepressant effect for fluoxetine that is significantly more marked than the effect produced by placebo’ (Rickels et al., 1986). Five additional articles contained mild spin (e.g. suggesting the treatment is at least numerically better than placebo). One article lacked an abstract, but the discussion section concluded that there was a ‘trend for efficacy’. Hence, only four (5%) of 77 published trials unambiguously reported that the treatment was not more effective than placebo in that particular trial (Fig. 1d). Compounding the problem, positive trials were cited three times as frequently as negative trials.


The cumulative impact of reporting and citation biases on the evidence base for antidepressants (de Vries, et al. 2018)

A New York Times write-up of the study by Aaron E. Carroll (“Congratulations, Your study went nowhere!)” points to several other studies that have revealed these kinds of distortions. It also makes the critical observation that because positive research tends to be cited much more frequently than negative research (citation bias), this distortion quickly proliferates through scientific literature:

A modeling study published in BMJ Open in 2014 showed that if a publication bias caused positive findings to be published at four times the rate of negative ones for a particular treatment, 90 percent of large meta-analyses would later conclude that the treatment worked when it actually didn’t.

Clearly this is a serious problem which calls the integrity of scientific research into question. In short, why should we accept ‘scientific evidence’ if that evidence is more the product of institutionalised bias rather than of open scientific enquiry.

Positive results bias is one of a number of problems in science that could be tackled by moving toward an open science approach. Open science can be defined as “the practice of carrying out scientific research in a completely transparent manner, and making the results of that research available to everyone” (Watson, 2015). The FDA require that all drug trials intended to help a treatment to gain FDA approval must be preregistered with them and the results published regardless of the outcome. This is a form of open science because it is opening up scientific research for anyone to access. It seems obvious that the more that science is moved out into the open, the less incentives there will be for things like positive results bias to distort it.

 

This post has first been published on Afroscientific.com

About

Afriscitech, all science throughout all Africa.

Follow the news about scientific and technological research in Africa and by Africans.

 

Who are we?

Afriscitech.com is published by Coopetic.

- Luc Allemand, Editor in Chief

Cookies settings

×

Functional Cookies

This site uses cookies to ensure its proper functioning, which cannot be disabled from our systems. We do not use them for advertising purposes. If these cookies are blocked, some parts of the site will not work.

Measure of audience

This website uses cookies to measure and analyse audiences, such as Google Analytics and Google Ads, in order to evaluate and improve our website.

Interactive Content

This site uses third-party components, such as ReCAPTCHA, Google Maps, MailChimp or Calameo, which may store cookies on your machine. If you decide to block a component, the content will not be displayed

Social Networks / Videos

Social network and video plug-ins, which use cookies, are available on this website. They make it possible to improve the user-friendliness and promotion of the site through various social interactions.

Other cookies

This Joomla CMS uses a number of cookies to manage user sessions, for example.

Search