Cookies help us deliver our services. By using our services, you agree to our use of cookies. More information

Difference between revisions of "Reproducibility crisis"

From Bioblast
(3 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{MitoPedia
{{MitoPedia
|description=An experiment or study is ''reproducible'' or ''replicable'' when subsequent experiments confirm the results. However, we can define different types of reproducibility depending on the conditions that we use to replicate the previous work or in the information available. Our aim is to focus mostly on two different kinds<sup>2</sup>:
|description=An experiment or study is ''reproducible'' or ''replicable'' when subsequent experiments confirm the results. This is [[research |re-search]]. However, we can define different types of reproducibility depending on the conditions that we use to replicate the previous work or in the information available. Our aim is to focus mostly on two different kinds<sup>1</sup>:


'''1. Direct:''' Is when we obtaining the same results using the same experimental conditions, materials, and methods as described in the original experiment. This would be the ideal reproducibility of an experiment however, it requires a very accurate description of how the original experiment was performed. Some journals are trying to resolve the '''reproducibility crisis''' improving the rigor and the excellence on the reported methods and results (e.g. [https://www.cell.com/star-authors-guide STAR Methods in Cell Press]).
'''1. Direct:''' Is when we obtaining the same results using the same experimental conditions, materials, and methods as described in the original experiment. This would be the ideal reproducibility of an experiment however, it requires a very accurate description of how the original experiment was performed. Some journals are trying to resolve the '''reproducibility crisis''' improving the rigor and the excellence on the reported methods and results (e.g. [https://www.cell.com/star-authors-guide STAR Methods in Cell Press]).
Line 6: Line 6:
'''2. Systematical:''' Refers to obtaining the same results, but under different conditions; for example, using another cell line or mouse strain, or inhibiting a gene pharmacologically instead of genetically. This open the door to subsequent studies to find the conditions under which an initial finding holds.
'''2. Systematical:''' Refers to obtaining the same results, but under different conditions; for example, using another cell line or mouse strain, or inhibiting a gene pharmacologically instead of genetically. This open the door to subsequent studies to find the conditions under which an initial finding holds.


But, is there a '''reproducibility crisis'''? According to a survey conducted by ''Nature'' of 1,576 researchers, "52% agree that there is a significant 'crisis' of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature" (Baker 2016). Chemistry and biology are the subjects with the highest share of failed attempts of reproduction of results.  
But, is there a '''reproducibility crisis'''? According to a survey conducted by ''Nature<sup>2</sup>'' of 1,576 researchers, "52% agree that there is a significant 'crisis' of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature" (Baker 2016). Chemistry and biology are the subjects with the highest share of failed attempts of reproduction of results.  


Asking the researchers for the causes of this inability to reproduce published results, the top three answers are:
Asking the researchers for the causes of this inability to reproduce published results, the top three answers are:
Line 19: Line 19:




|info=::1. [https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 Baker M (2016) 1,500 scientists lift the lid on reproducibility. Survey sheds light on the ‘crisis’ rocking research. Nature 533:452–4.]
|info=::1. Stanley E Lazic (2016) Experimental design for laboratory biologists. Maximizing information and improving reproducibility. Cambridge University Press.
::2. Experimental Design for Laboratory Biologists. Maximizing Information and Improving Reproducibility. Stanley E Lazic. Cambridge University Press.2016
::2. [https://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970 Baker M (2016) 1,500 scientists lift the lid on reproducibility. Survey sheds light on the ‘crisis’ rocking research. Nature 533:452–4.]
}}
}}


Line 26: Line 26:
== Further links ==
== Further links ==


::::* Gnaiger E (2019) Editorial: A vision on preprints for mitochondrial physiology and bioenergetics. MitoFit Preprint Arch doi:10.26124/mitofit:190002.v2. - [[Gnaiger 2019 MitoFit Preprint Arch Editorial |»MitoFit Preprint Arch«]]
::::* [https://www.nature.com/articles/s41467-018-06012-8 Reproducibility: let’s get it right from the start. Nature Communications 9: Article number 3716 (2018)]  
::::* [https://www.nature.com/articles/s41467-018-06012-8 Reproducibility: let’s get it right from the start. Nature Communications 9: Article number 3716 (2018)]  



Revision as of 07:05, 8 May 2019


high-resolution terminology - matching measurements at high-resolution


Reproducibility crisis

Description

An experiment or study is reproducible or replicable when subsequent experiments confirm the results. This is re-search. However, we can define different types of reproducibility depending on the conditions that we use to replicate the previous work or in the information available. Our aim is to focus mostly on two different kinds1:

1. Direct: Is when we obtaining the same results using the same experimental conditions, materials, and methods as described in the original experiment. This would be the ideal reproducibility of an experiment however, it requires a very accurate description of how the original experiment was performed. Some journals are trying to resolve the reproducibility crisis improving the rigor and the excellence on the reported methods and results (e.g. STAR Methods in Cell Press).

2. Systematical: Refers to obtaining the same results, but under different conditions; for example, using another cell line or mouse strain, or inhibiting a gene pharmacologically instead of genetically. This open the door to subsequent studies to find the conditions under which an initial finding holds.

But, is there a reproducibility crisis? According to a survey conducted by Nature2 of 1,576 researchers, "52% agree that there is a significant 'crisis' of reproducibility, less than 31% think that failure to reproduce published results means that the result is probably wrong, and most say that they still trust the published literature" (Baker 2016). Chemistry and biology are the subjects with the highest share of failed attempts of reproduction of results.

Asking the researchers for the causes of this inability to reproduce published results, the top three answers are:

  • Selective reporting
  • Publication pressure
  • Low statistical power and poor analysis

The top three mentioned countermeasures are:

  • Better understanding of statistics
  • Better mentoring and supervision
  • More robust design


Reference:

1. Stanley E Lazic (2016) Experimental design for laboratory biologists. Maximizing information and improving reproducibility. Cambridge University Press.
2. Baker M (2016) 1,500 scientists lift the lid on reproducibility. Survey sheds light on the ‘crisis’ rocking research. Nature 533:452–4.


Further links



Solve the reproducibility crisis

While it is probably impossible to fully prevent human self-deception and inadequate command of statistical methods, what we can do is minimize sources of error connected to the instrumental equipment and its handling:
  • Select instrumental equipment for which appropriate specifications are available.
  • Have yourself trained on your equipment and make sure you know what you (both, you and the device you operate) are doing in each step of your experiment.
  • Avoid black-box performance of software.
  • Same for data analysis: get trained on analysis software. In the best case, use software that comes with your instrument in order to minimize errors during data transfer and translation.
  • An Open Access policy fosters the establishment of an error culture and a culture of transparence in science. In this way, Open Access - as manifested in the Bioblast website (see Gentle Science - contributes to solving the reproducibility crisis.
  • Methods: Identify the methods, apparatus (manufacturer's name and address in parentheses), and procedures in sufficient detail to allow other workers to reproduce the results. Give references to established methods. - Quoted from International Committee of Medical Journal Editors.


Further links

  • Validation by THE SCIENCE EXCHANGE NETWORK
» Validating key experimental results via independent replication
» Reproducibility Initiative


MitoPedia concepts: "MitoFit Quality Control System" is not in the list (MiP concept, Respiratory state, Respiratory control ratio, SUIT concept, SUIT protocol, SUIT A, SUIT B, SUIT C, SUIT state, Recommended, ...) of allowed values for the "MitoPedia concept" property. MitoFit Quality Control System"MitoFit Quality Control System" is not in the list (Enzyme, Medium, Inhibitor, Substrate and metabolite, Uncoupler, Sample preparation, Permeabilization agent, EAGLE, MitoGlobal Organizations, MitoGlobal Centres, ...) of allowed values for the "MitoPedia topic" property.