What is an advantage of replicability in experiments?

Replication of a study is repeating a study’s procedure and observing if the prior findings repeat in similar conditions. A study is replicated when the results of original study are as closely related to the newly collected data.

What is the Main Purpose of Replication Studies?

The credibility of scientific studies is established only if it is replicable under similar or closely related conditions. Findings collected from such studies gives greater validity to the originally conducted research. Furthermore, it means that the original study is more likely to be generalized for larger applications and future research scope.

What are the Types of Replication Studies?

Replication studies are broadly classified as:

1. Exact or Direct replications– Direct replication is the repetition of an experimental procedure to the exact degree as possible. It means that exactly same equipment, material, stimuli, design and statistical analysis should be used.
2. Conceptual replications– Conceptual replication is when research is conducted by using different methods to repeat the original study. Despite difference in methods, the new data is similar to the original study findings.

An Unfair Reputation?

A replication study attempts to validate the findings of a prior piece of research. By doing so, that prior research is confirmed as being both accurate and broadly applicable, since the replication study typically changes one or more variables of the original study, such as sample population, industry sector, etc.

From the perspective of contributing to scientific research, replication studies are important for the continued progress of science. Without validation, how do future researchers know whether to build on the findings of that original work?

However, from the perspective of the individual researcher’s career, replication studies do not carry the same weight as an original piece of work. Research Journals aren’t eager to publish such work because it doesn’t qualify as new research, and is therefore of less interest to their readers.

The Challenges of a Replication Study

Being able to replicate a simple study from a decade ago and observe how things may have changed over the last ten years, has saved the fortunes of more than one doctoral student struggling to find a research topic. However, replicating more recent research, especially projects making controversial claims, can prove to be extremely challenging.

If the original research team is unwilling to share all of their data, any colleagues looking to validate their work through replication are already placed at a disadvantage. Blaming the protocol of the replication study rather than reexamining the original results can easily dismiss any discrepancies from the original findings. For this reason, controversial studies can stand for years without retraction, even with multiple failed validation studies.

A Research Quandary

Operating in a highly competitive publish or perish environment, new researchers are presented with a quandary here. Research Supervisors encourage the use of replication studies to both provide a valuable contribution of validation to science, and to expose their students to multiple research methodologies.

However, those new researchers are eager to build their academic track records and therefore want to do research that will get published. Since replication studies typically don’t get published, that doesn’t help their track records.

Time to Overcome an Industry Bias

Even the best scientists make mistakes, and replication studies provide a valuable contribution in catching such mistakes before flawed studies get too widely dispersed. No one disputes the need for validation, especially in this climate of highly competitive research that is expected to produce maximum results as quickly as possible.

However, we are currently trapped in a vicious cycle of “prestige-chasing” that has to change. The less chance a proposed study has of being published in a prestigious journal, the less likely it is that the study will receive funding. Even if it does receive funding, researchers are unlikely to compete to participate in the study if they don’t see a chance of getting it published.

The days of open access journals and pay-for-print journals that charge article processing fees (APFs), have rapidly expanded the number of journals out there. Unless some of them start setting aside space for replication studies, the resources assigned to research validation will continue to decline, and flawed research that cannot be validated will be allowed to persist without challenge or retraction.

Reproducibility and Replicability have meant different things to different disciplines. While most individuals are aware that disciplines have identified issues in the areas of reproducibility and replicability, the disciplines themselves have not been in agreement regarding the common description and definition of these two terms. If you want to start a heated argument, add repeatability to the list!

For a brief discussion of this topic, please see: Plesser, Hans E. "Reproducibility vs. Replicability: A Brief History of a Confused Terminology," in Frontiers of Neuroinformatics, 18 January 2018. https://doi.org/10.3389/fninf.2017.00076

Working definitions are as follows though we recognize that the terms have been used interchangeably among disciplines.

• Replicability (same experimental setup): The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts. (Repeatability often refers to the same concept but with the same team of researchers).
• Reproducibility (Different team, different experimental setup): The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

Of course, "sameness" of procedures and results can be further defined in different disciplines. For example...

• The "same experimental procedure” may mean – downloading the original software and data and run it; downloading the original software, compiling it for a different machine and running it with original data; downloading software that carries out the same operations and applying  it to the original data; reading a paper producing a new implementation of the algorithms and running it on the original data; or running any of the above on a refined or updated data set
• The “Same results” could mean – identical output at the bit level; exactly the same numbers; exactly the same numbers on a similar machine; numbers that are within some bound of error; or outputs that share certain characteristics

And reproducibility can come in a number of flavors -- Computational reproducibility and transparency, Scientific reproducibility and transparency, Computational correctness and evidence, and Statistical reproducibility

Reproducibility and replicability do not refer to one standard set of guidelines but rather are set through individual research communities and organizational standards.

What are the advantages of replication in research?

What is replicability and why is it important?

Does replicability improve reliability?

Why is replicating important?