Reproducibility: Has Cancer Biology Failed beyond Repair?

Cancer is the second major cause of death worldwide, and its burden of disease increases over time. Discovering, developing, and implementing useful cancer interventions is a top priority, and pressure is high on regulatory agencies to accelerate cancer-related approvals. However, 90% to 95% of cancer drugs that enter clinical trials are not approved eventually. Worse, among the drugs licensed in the recent past, the accumulated evidence has been disappointing. These interventions typically improve surrogate endpoints but achieve minimal or no survival improvements. This is in contrast to earlier, obvious major successes achieved in oncology (e.g., in treating pediatric malignancies). Cancer biomarkers research similarly produces many thousands of publications annually but rarely translates to major tools saving lives. Why are we failing? The early steps in the translational pipeline of cancer research may be broken. Typically this research involves animals and/or tissue models. Cancer biology comprises huge networks of scientific activity involving many hard-working, well-intentioned scientists. The most exciting papers with the biggest promises easily become citation classics, among the most-cited across all science. But then, most of these papers go nowhere—no lives are saved. The Reproducibility Project: Cancer Biology (RPCB) (1–3) has helped us to better understand this failure. RPCB set out to perform reproducibility checks for 193 cancer biology experiments included in 50 highly cited papers published from 2010 to 2012. After 8 years of attempted replication work by many competent scientists, communicating with the original investigators, and following the best research practices possible with rigorous preregistration and high standards of external peer-review, the RPCB team could report some results for only 50 of 193 experiments. The oxymoron is mind-boggling: drug companies seek licensing of their drugs by regulatory agencies within months (if not weeks); conversely, merely repeating even the biological science that underlies such work took an average of 3 years per each experiment. For the 143 abandoned experiments, the confessed description of the failure by the RPCB team is sobering. First, only 4/193 experiments presented in their publication the minimum statistical information necessary to compute effect sizes and perform some power calculations for designing a reproducibility experiment. Statistical innumeracy is extremely prevalent across animal and in vitro research—not just cancer-related. The RPCB team then asked the original investigators, but they could retrieve this essential information less than one third of the time. Second, none of the 193 experiments were described in sufficient detail in the Methods of the published papers to allow someone to repeat them. Once again, the RPCB team reached out to the primary investigators for help, but only 41% of the investigators were very helpful. As many were very minimally helpful or did not even reply. Third, among the experiments that started, two thirds required modifications at some point, and most modifications could not be implemented, so the effort was abandoned. Of the 50 experiments (across the 23 papers) that could eventually be pursued, the RPCB team prespecified 7 different ways to read the results. If 7 ways seem a lot, in reality, there are many more. A growing methodological literature debates how to claim that replication was successful or not. These different methods have different thresholds of success, and the exact readout of the whole RPCB differs accordingly. However, consistent patterns emerged, regardless of “success” definitions. When the highly cited paper had found something, the reproducibility effort typically found nothing or found a far weaker signal. Conversely, when the highly cited paper admitted null results, almost always the reproducibility effort agreed. Furthermore, animal research and small effect sizes almost always failed to replicate. Perhaps the most relevant way to read the RPCB results is through effect sizes: on average, 85% of the effect originally detected in the highly cited study evaporated in the replication. Would one launch a commercially oriented development effort—investing almost $1 billion to develop a new drug, biologic, or anything—if one knew that, on average, the expected effect is only 15% of what was claimed originally? The answer depends on the exact topic, and perhaps some companies still exist that wish to take high risks (and almost certainly waste $1 billion). However, the common sense answer would be, generally, no. Another reality check might be the following: Has any of the 50 highly cited papers resulted in something that has demonstrably saved many lives by now, 10 to 12 years after its publication? I would not be surprised if the answer is also, generally, no. However, an in-depth assessment is warranted, and I would be extremely happy if my pessimistic suspicion is amply contradicted. It is important to document whether replication failure aligns or not with translational failure. The debated “replication crisis” has led to a healthy massive response of the scientific community and of other stakeholders involved in research (4). Deficiencies in planning, designing, sharing, repeating, reviewing, publishing, disseminating, and rewarding research have been identified. Many efforts have tried to fix the identified problems and improve the record of successes. The RPCB team noticed that most published experiments suffered from problems that should have been easy to fix preemptively; for example, animal experiments typically lacked blinding and randomization, while both blinding and randomization can be easily and routinely applied to any experiment. Sample sizes can become larger, designs more rigorous, data and code more widely shared, and so forth. However, realistically, how much can all these amendments improve the performance of the field? Admittedly, one may still argue that the field needs no improvement. Many scientific fields that go through the traumatic experience of a replication scrutiny with bleak results often go into a defensive mode. One can readily argue that the replications may be wrong or the replications may be correct but looking at something different than the original. Indeed, these are definite possibilities. Moreover, they represent a “politically correct” stance in situations where investigators feel threatened by the “negative” replication studies. One should reassure everyone that replication is not a police interrogation. No one is being blamed (overt fraud aside, which is hopefully uncommon). Failure rate close to 100% may well be the norm in some exciting disciplines. Only boring research fields looking at already strongly established (thus uninteresting) signals may have extremely high replication rates. This being said, some serious soul-searching questions should be posed for cancer biology. The translational work of this discipline is certainly important given the huge burden of disease. One certainly cannot and should not test all new drugs, biologics, diagnostics, and other technologies directly on humans, since the failure rate is extremely high. Animal work and other basic translational efforts are essential to narrow down candidates. However, it seems that one cannot narrow down efficiently with the current research agenda. Kimmelman et al. (5) calculate sobering likelihood ratios for cancer biology research as a diagnostic tool separating what might eventually work vs what might not. In my opinion, the lower estimates (e.g., positive likelihood ratios of 2-4) are more likely to reflect current reality: a “positive” finding improves the chances of eventual “success” only 2- to 4-fold, The better performance of negative likelihood ratios (a “negative” finding largely excludes the possibility of eventual “success”) is nice but does not alleviate the major problem. We still have zillions of promising leads and most lead nowhere. Lives are not saved. A massive shift in the reward and incentives system may optimize the positive likelihood ratios. There is plenty of low-hanging fruit to improve research practices, as discussed previously. However, most of these improvements may shrink the number and proportion of prestigious papers, highly funded work, and glamorous careers. Researchers should be rewarded for doing the right thing, which almost always reduces the chances of becoming successful according to these old-fashioned criteria. Part of this reform may be effectuated within academia, but the transformative process has been difficult to materialize to date in academic environments. A more drastic solution would be to largely remove translational cancer biology research with deliverables from the academic circuit and pass it over to private entrepreneurs. The industry may have every reason to perform reliable, reproducible cancer biology research: they would hate waste $1 billion investments on bubbles. Not surprisingly, the first empirical demonstrations about the lack of reproducibility of academic-initiated cancer biology came from investigators at Beyer and Amgen. Here, one has to make sure that such a shift from academia to industry will not hurt basic science where there is no clear deliverable but plain curiosity, unhindered exploration, and high-risk innovation. Research funding for genuinely basic cancer biology and foundational biomedical research in general needs to increase, not shrink. However, currently the NIH budget and the budget of many other funding agencies is absorbed by translational efforts that are not really foundational but have clear applied goals. This huge research agenda has practically diminished the public funding for genuinely innovative basic science. It has also almost eliminated public funding for evidence-based, clinical, and implementation research that is badly needed—because industry products of applied, translational efforts should be assessed rigorously by industry-independent researchers. Salvaging cancer biology (and biomedical research in general) may need some rethinking of who does what, who replicates what, and who rewards what. Author Contributions:All authors confirmed they have contributed to the intellectual content of this paper and have met the following 4 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved. Authors' Disclosures or Potential Conflicts of Interest:No authors declared any potential conflicts of interest.