You are here
August 10, 2023
Addressing Rigor in Scientific Studies
Devon C. Crawford, Ph.D., Program Director, Office of Research Quality, NIH鈥檚 National Institute of Neurological Disorders and Stroke
Science communication is rapidly evolving. The growing use of preprints and the sheer number of published studies make it increasingly difficult to determine which findings are worthy of attention. Not all scientific studies are created equal. Communicators need to discern which are reputable in order to know what to convey to their target audience. Inaccurate or untrustworthy information can have dire consequences, so it is important to understand how to assess whether studies have robust findings and how to communicate this to audiences.
Science communicators need to describe the major conclusions from a study, along with its implications for future research and public health practice, without overstating the results. Science is a continual process of updating knowledge that is conditional on how the results were obtained; it is not a series of discovered 鈥渇acts.鈥 All scientific conclusions are subject to interpretation, and all have some degree of uncertainty. Responsible science communicators will report important details of a study: the number of subjects, species involved, techniques used, major outcomes, and caveats. But, even this level of reporting does not provide enough information to know how much to trust the results.
As for , the of a study are key for gleaning the robustness of its results. This includes the design, implementation, analysis, and interpretation of experiments. If a study鈥檚 validity isn鈥檛 known, the rest is moot.
How does one know if a study is rigorous? And how can this be communicated to broad audiences? A single person can鈥檛 keep up with all of the limitations of every scientific approach, and even savvy readers of original research articles need to beware of within a given study. Fortunately, there are some generally agreed-upon that apply across fields and methods.
Transparent publications follow established guidelines to ensure that important research practices are reported. These include the for clinical trials, for animal studies, and for systematic reviews. It is difficult to assess the rigor and robustness of studies that do not fully follow these guidelines. Yet, .
Given the limited adherence to the lengthy lists contained in these guidelines, NIH鈥檚 National Institute of Neurological Disorders and Stroke (NINDS) held a workshop in 2012 to identify ways to enhance rigor-related transparency. The result was a list of that should be reported for every relevant research study, from basic research to clinical trials:
- Sample size estimation: how the sample size, such as the number of participants, was chosen before the study began, ideally via statistical extrapolation from prior studies.
- Blinding/Masking: how experimenters ensured that none of those involved in the study knew which samples or groups received the intervention being tested.
- Randomization: how experimenters selected groups for treatment or data analysis so that all had the same chance of undergoing an intervention.
- Data handing: how experimenters planned in advance to manage missing data, outliers, what data to exclude from analysis, and when to stop data collection.
Without these practices, studies are likely to be at high risk for unconscious biases that can lead to incorrect conclusions. Depending on the goals of the study, not all these items may be relevant. However, it is still important to report whether they were relevant and used for a particular study to make it clear which measures were taken to reduce bias. Other that should be considered include transparent reporting of outcome measures, using appropriate control groups, defining clear measures of uncertainty, and addressing study limitations such as possible confounding factors. If the study is exploratory, if the above practices are not reported or performed, or if other relevant reporting guidelines have not been followed, the results should be interpreted as preliminary or tentative, and any conclusions should be communicated accordingly.
Importantly, each study must also be put within the appropriate context of the wider scientific landscape. Are the results consistent with previous studies? Were those previous studies rigorously performed and transparently reported? Is there a chance that these results were only published because of the exciting result and not because of the rigor of the methods? Could there be similar or more rigorous studies out there that were not published or publicized because they did not get this exciting outcome? In other words, how surprising was this result? The more surprising, the more robust the evidence needs to be to support it.
Science communicators can鈥檛 be expected to assess all these questions themselves. Communicators should ask the study investigators and other experts in the field for support in verifying important design elements of a particular study. If information is missing, why hasn鈥檛 it been addressed? Without this information, the finding cannot be properly interpreted.
Communicating science accurately and responsibly is a balance between engaging the audience and providing enough important details. NIH鈥檚 NINDS explicitly emphasizes that rigor and transparency 鈥.鈥 They shouldn鈥檛 be left out of communications about research. Addressing a study鈥檚 rigor, transparency, and robustness not only provides important information about reliability, but it also signals to the audience that context, and by extension the ongoing process, is central to science.