Science faces a crisis of confidence owing to its politicization and the replication crisis. The latter, which intersects with concerns about scientific fraud, is the subject of this post, a matter on which I’ve previously commented. (OK, ranted.)
Part of the impetus was from a study by Amgen, summarized in Wikipedia as follows:
Out of 49 medical studies from 1990–2003 with more than 1000 citations, 45 claimed that the studied therapy was effective. Out of these studies, 16% were contradicted by subsequent studies, 16% had found stronger effects than did subsequent studies, 44% were replicated, and 24% remained largely unchallenged.[59] The US Food and Drug Administration in 1977–1990 found flaws in 10–20% of medical studies.[60] In a paper published in 2012, Glenn Begley, a biotech consultant working at Amgen, and Lee Ellis, at the University of Texas, argued that only 11% of the pre-clinical cancer studies could be replicated.[61][62]
To try to put some numbers to my visceral impressions, and thereby to test my conjectures regarding dodgy results I went over to Retraction Watch, which features a database of papers that have been retracted by the authors/journals or otherwise had their validity called into question.
One can rummage around in that database for hours, so I narrowed the search to “concerns/issues about results,” which yielded 562 hits.
The requisite data to normalize for the number of papers published in each field are not to hand, but with that caveat these data bear out the impression that the biomedical literature has some real problems.
Where, geographically, do these problems come from? Querying the database for papers that raised concerns/issues with results (regardless of whether from physical, life, or medical sciences) yielded these results, again not normalized for the number of papers published in each country:
Without information regarding the number of papers published, it’s impossible to draw hard conclusions, but I would imagine that there are probably a lot more papers published in the U.S. than in China, and yet the two are in the same ball park regarding questionable results.
Interestingly, the database includes data regarding some of the more dubious areas of “science.” For this search I included outright scientific fraud, and found effectively none for any of these fields, although a similar search the physical, biological, and health sciences all returned a number of hits. Once again, the caveat regarding total number of papers published applies, this time probably a forteriori.
| Subject | falsification/fabrication of results (papers) | concerns about results |
| Climate change | 0 | 2 |
| Sociology | 0 | 17 |
| Alternative medicine | 0 | 14 |
| Psychology | 1 | 53 |
For comparison:
| Subject | falsification/fabrication of results (papers) | concerns about results |
| Physics | 4 | 14 |
| Chemistry | 15 | 21 |
| Biology | 65 | 222 |
| Medicine | 70 | 305 |
My conclusion, subject to the caveat above: the biomedical literature does in fact have grievous problems, and China is punching above its weight in generating lousy results. Last, the Wild West of “scientific research” (climate change, sociology, alternative medicine, and psychology) I suspect may not have fraudulent papers because results in those areas are so squishy that it is impossible to falsify pretty much any assertion made in those literatures.
