Hello there bloggies! Welcome to Science Sunday! Before getting into this blog, I’ve just realized that I can schedule future blog posts as opposed to posting them in real time. This is going to be so helpful staying on schedule during some of my travels! Anyways, on to Science Sunday! Today, I wanted to discuss one of the hallmarks of scientific research: reproducibility.
First of all, what is reproducibility? To me at least, reproducibility has many different levels of definitions in research.
The first place reproducibility comes into play is with you personally. Can you repeat your experiment using the same protocols and materials and get the same results? If you can’t reproduce your results, then they simply can’t be trusted and you don’t have to worry about the other levels of reproducibility. There’s nothing to reproduce.
If you can replicate your experiments results yourself in your lab, then we can move onto the next level on reproducibility. Can others in your lab and in the field, following the same procedures and using the same materials, get similar results to your experiments? Unless you are Harry Potter following the instructions of the Half-Blood Prince, your labmates and colleagues in your field should be able to follow your experimental and analytical protocols and get the same outcomes. If they can’t, then it is incredibly difficult to trust the results of your experiments. However, if your colleagues and competitors can reproduce them, then you can have even more confidence that your results and the conclusions drawn from them are valid.
It feels so incredibly fantastic (and also relieving…) to have your results validated by others, including any not so nice competitors you may have in the field. But, there is another level of reproducibility. This time, however, it doesn’t involve you or someone else simply repeating your experiments or analyses. There are so many different experimental approaches to investigate a scientific question. If these various approaches begin to converge on the same conclusions as your initial experiments, then that’s even better evidence that your results are valid and not a result of inherent biases to your approach. You may actually be moving toward a substantial, accurate contribution to science!
Now, we have a better understanding of what reproducibility is. But, who should evaluate reproducibility? When should reproducibility be examined? How? The simple answers are that everyone should be examining the reproducibility of experiments/analyses at all times!
First of all, you should be the biggest evaluator of your own experiment’s reproducibility. Be skeptical and thorough with your work. Do important experiments multiple times. Keep good records. Your scientific reputation, objectiveness, and accuracy are so very important to career in research. Do your best to keep it intact.
After you are convinced of the reproducibility of your work, your labmates and most importantly the head of lab should vet these results either by repeating the experiments/analyses themselves or by thoroughly reviewing them in lab meeting or some other forum. They should be asking you the tough questions of your approach, methods, and the conclusions you are drawing from the results. The toughest questions should come here, not later from the scientific community.
Once you have convinced your lab and collaborators that your experiments are valid and reproducible, then it is on to the peer review process. It is the job of these referees to objectively evaluate the methods used to address the scientific question and to determine whether the results presented in a manuscript support the conclusions made. These reviewers should have access to all available data (e.g., gene expression deposition in GEO) and the tools used to analyze these data.
The final judges of reproducibility are the scientific community. Everyone who reads a scientific manuscript should ask themselves whether the methods used are sound and if they agree with the authors interpretation of the results. They should ask themselves whether these results pass the bs test. Are they too good to be true? Do the figures and tables look right?
The authors still have responsibilities here. When ethically possible, raw data including gene expression and screening assays should be made available to other investigators as well as new software and the scripts used in analyzing the data. Post-publication review is becoming a more important and more visible part of the scientific community through several different mediums including Twitter, PubPeer, and PubMed Commons. Recently, there was a fairly notable instance of a post-publication review that went viral in the scientific community discussed here.
Replication and reproducibility of results are major parts of the scientific process. They should be taken into consideration before, during, and after each set of experiments and analyses scientists do. Only then can the community move forward the most accurate results and conclusions possible.
Thank you all for visiting Science Sunday! How does reproducibility influence your work in the lab? What do you think about how the scientific community evaluates reproducibility? Comment below or on twitter @DrFsThoughts.
See you all on Trainee Tuesday!