Ioannidis 2005 PLoS Med
|Ioannidis JP (2005) Why most published research findings are false. PLoS Med 2005 Aug;2(8):e124.|
Abstract: There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
Labels: MiParea: Instruments;methods
Selected text quotes
- “Negative” research is also very useful. “Negative” is actually a misnomer, and the misinterpretation is widespread. However, here we will target relationships that investigators claim exist, rather than null findings.
- Bias: the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced. .. Bias can entail manipulation in the analysis or reporting of findings. Selective or distorted reporting is a typical form of such bias. .. Traditionally, investigators have viewed large and highly significant effects with excitement, as signs of important discoveries. Too large and too highly significant effects may actually be more likely to be signs of large bias in most fields of modern research.
- In contrast is chance variability: causes some findings to be false by chance even though the study design, data, analysis, and presentation are perfect.
- Measurement error has decreased with technological advances in the molecular era and investigators are becoming increasingly sophisticated about their data.
- Independent teams: As research efforts are globalized, it is practically the rule that several research teams, often dozens of them, may probe the same or similar questions. Unfortunately, in some areas, the prevailing mentality until now has been to focus on isolated discoveries by single teams and interpret research experiments in isolation. .. It is misleading to emphasize the statistically significant findings of any single team. What matters is the totality of the evidence.
- Sample size: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true. .. Other factors being equal, research findings are more likely true in scientific fields that undertake large studies, such as randomized controlled trials in cardiology (several thousand subjects randomized) than in scientific fields with small studies, such as most research of molecular predictors (sample sizes 100-fold smaller). .. Large-scale evidence is impossible to obtain for all of the millions and trillions of research questions posed in current research.
- Effects: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true. .. Modern epidemiology is increasingly obliged to target smaller effect sizes. .. If the true effect sizes are very small in a scientific field, this field is likely to be plagued by almost ubiquitous false positive claims.
- Relationships: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true.
- Research design: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true. .. Adherence to common standards is likely to increase the proportion of true findings. The same applies to outcomes. True findings may be more common when outcomes are unequivocal and universally agreed (e.g., death) rather than when multifarious outcomes are devised. .. Fields that use commonly agreed, stereotyped analytical methods may yield a larger proportion of true findings than fields where analytical methods are still under experimentation.
- Conflict of interest: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true. .. Scientists in a given field may be prejudiced purely because of their belief in a scientific theory or commitment to their own findings. Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure. .. Prestigious investigators may suppress via the peer review process the appearance and dissemination of findings that refute their findings, thus condemning their field to perpetuate false dogma.
- Mainstream: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. .. With many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritize on pursuing and disseminating its most impressive “positive” results.
- Null fields: Fields with absolutely no yield of true scientific information.
- Improvements: Diminishing bias through enhanced research standards and curtailing of prejudices may .. help. However, this may require a change in scientific mentality that might be difficult to achieve.
- In some research designs, efforts may also be more successful with upfront registration of studies. Registration would pose a challenge for hypothesis-generating research.
- Some kind of registration or networking of data collections of investigators within fields may be more feasible than registration of each and every hypothesis-generating experiment.
- The principles of developing and adhering to a protocol could be more widely borrowed from randomized controlled trials.
- Before running an experiment, investigators should consider what they believe the chances are that they are testing a true rather than a non-true relationship.
- Whenever ethically acceptable, large studies with minimal bias should be performed on research findings that are considered relatively established, to see how often they are indeed confirmed.
- Usually it is impossible to decipher how much data dredging by the reporting authors or other research teams has preceded a reported research finding. Even if determining this were feasible, this would not inform us about the pre-study odds.