Afroscientific

The website catalogofbias.org summarises positive results bias as "the tendency to submit, accept and publish positive results rather than non-significant or negative results."

It continues: “positive results bias occurs because a considerable amount of research evidence goes unpublished, which contains more negative or null results than positive ones. This leads to spurious claims and overestimation of the results of systematic reviews and can also be considered unethical. Non-publication of results can also lead to research wastage as researchers may unnecessarily repeat studies because the results are unpublished.”

An August 2018 study in Psychological Medicine demonstrated the positive results bias in trials on antidepressant drugs. In “The cumulative effect of reporting and citation biases on the apparent efficacy of treatments: the case of depression”, the researchers looked at 105 drug trials that had been registered with the US Food and Drug Administration. They explain that “pharmaceutical companies must preregister all trials they intend to use to obtain FDA approval; hence, trials with non-significant results, even if unpublished, are still accessible.”

This is an important point which we’ll return to. But the main findings of the study make for startling reading:

“Of 105 antidepressant trials, 53 (50%) trials were considered positive by the FDA and 52 (50%) were considered negative or questionable (Fig. 1a). While all but one of the positive trials (98%) were published, only 25 (48%) of the negative trials were published. Hence, 77 trials were published, of which 25 (32%) were negative (Fig. 1b). Ten negative trials, however, became ‘positive’ in the published literature, by omitting unfavorable outcomes or switching the status of the primary and secondary outcomes (Fig. 1c). Without access to the FDA reviews, it would not have been possible to conclude that these trials, when analyzed according to protocol, were not positive. Among the remaining 15 (19%) negative trials, five were published with spin in the abstract (i.e. concluding that the treatment was effective). For instance, one article reported non-significant results for the primary outcome (p = 0.10), yet concluded that the trial ‘demonstrates an antidepressant effect for fluoxetine that is significantly more marked than the effect produced by placebo’ (Rickels et al., 1986). Five additional articles contained mild spin (e.g. suggesting the treatment is at least numerically better than placebo). One article lacked an abstract, but the discussion section concluded that there was a ‘trend for efficacy’. Hence, only four (5%) of 77 published trials unambiguously reported that the treatment was not more effective than placebo in that particular trial (Fig. 1d). Compounding the problem, positive trials were cited three times as frequently as negative trials.”

The cumulative impact of reporting and citation biases on the evidence base for antidepressants (de Vries, et al. 2018)

A New York Times write-up of the study by Aaron E. Carroll (“Congratulations, Your study went nowhere!)” points to several other studies that have revealed these kinds of distortions. It also makes the critical observation that because positive research tends to be cited much more frequently than negative research (citation bias), this distortion quickly proliferates through scientific literature:

“A modeling study published in BMJ Open in 2014 showed that if a publication bias caused positive findings to be published at four times the rate of negative ones for a particular treatment, 90 percent of large meta-analyses would later conclude that the treatment worked when it actually didn’t.”

Clearly this is a serious problem which calls the integrity of scientific research into question. In short, why should we accept ‘scientific evidence’ if that evidence is more the product of institutionalised bias rather than of open scientific enquiry.

Positive results bias is one of a number of problems in science that could be tackled by moving toward an open science approach. Open science can be defined as “the practice of carrying out scientific research in a completely transparent manner, and making the results of that research available to everyone” (Watson, 2015). The FDA require that all drug trials intended to help a treatment to gain FDA approval must be preregistered with them and the results published regardless of the outcome. This is a form of open science because it is opening up scientific research for anyone to access. It seems obvious that the more that science is moved out into the open, the less incentives there will be for things like positive results bias to distort it.

This post has first been published on Afroscientific.com

Clapperton Chakanetsa Mavhunga is Associate Professor at the Massachusetts Institute of Technology. More pertinently to this post, he is a Zimbabwean and a Shona. He has just published a new book which delves into the ways in which the Shona and other African people’s dealt with the Tse Tse Fly prior to the arrival of Europeans and colonialism.

This is a snippet of MIT’s overview of the book;

“Few animals are more problematic than the tiny African insect known to English speakers as the tsetse fly. This is the carrier of “sleeping sickness,” an often deadly neurological illness in humans, as well as a disease that has killed millions of cattle, reshaping the landscape and economy in some parts of the continent.

For generations, vedzimbahwe (the “Shona” people, builders of houses) and their African neighbors, assembled a significant store of ruzivo — knowledge — about mhesvi, their name for the tsetse fly. As MIT Associate Professor Clapperton Chakanetsa Mavhunga explains in a new book, this accumulation of local knowledge formed the basis for all subsequent efforts to control or destroy the tsetse fly and is an exemplary case of scientific knowledge being developed in Africa, by Africans.

“Ruzivo and practices based on it were the foundation of what became science and means and ways of tsetse control,” Mavhunga writes in “The Mobile Workshop: The Tsetse Fly and African Knowledge Production,” recently published by the MIT Press. However, he notes, Europeans nonetheless dismissed Africans as being “only good at creating and peddling myths and legends.”

In fact, Africans developed a diverse set of practices to combat mhesvi. For example, they used late-season forest burning to expose mhesvi to predators; moved herds through mhesvi-infested stretches at night while the insect was inactive; strategically located their settlements to neutralize the insect’s threat or turn it into a weapon against their human enemies; cleared bush and felled trees to create buffer zones between mhesvi-infested wildlife areas and human- and livestock-inhabited areas; and developed innoculations using live or dead mhesvi. Europeans appropriated many of these methods, or, at the very least, used their basic principles as starting points for what they then called “science.”  

Read the full article here.

This blogpost has been first published by Afroscientific.com