Protecting against researcher bias in secondary data analysis: challenges and potential solutions

Abstract

Analysis of secondary data sources (such as cohort studies, survey data, and administrative records) has the potential to provide answers to science and society’s most pressing questions. However, researcher biases can lead to questionable research practices in secondary data analysis, which can distort the evidence base. While pre-registration can help to protect against researcher biases, it presents challenges for secondary data analysis. In this article, we describe these challenges and propose novel solutions and alternative approaches. Proposed solutions include approaches to (1) address bias linked to prior knowledge of the data, (2) enable pre-registration of non-hypothesis-driven research, (3) help ensure that pre-registered analyses will be appropriate for the data, and (4) address difficulties arising from reduced analytic flexibility in pre-registration. For each solution, we provide guidance on implementation for researchers and data guardians. The adoption of these practices can help to protect against researcher bias in secondary data analysis, to improve the robustness of research based on existing data.

Cardiac sympathetic-vagal activity initiates a functional brain–body response to emotional arousal

“…The use of secondary, previously released, data raises the issue of challenges and problems with the preregistration process and the other standard polices to ensure good research practices. While, in a strict sense, preregistration has been developed and widely used for primary data, the use of secondary data poses several difficulties with standard preregistration as recently discussed ( 89 , 90 ). While a general agreement on the most correct approach is still lacking, a relevant suggestion is to declare the working hypothesis in advance ( 90 ).…”

“…While, in a strict sense, preregistration has been developed and widely used for primary data, the use of secondary data poses several difficulties with standard preregistration as recently discussed ( 89 , 90 ). While a general agreement on the most correct approach is still lacking, a relevant suggestion is to declare the working hypothesis in advance ( 90 ). Such statements have been done in previous scientific peer-reviewed publications from the authors, in which it is stated that ascending afferents to the brain play a role in shaping perception, consciousness, and emotions ( 43 , 46 , 52 , 61 , 91 ).…”

“…Moreover, the use of public data and the availability of our analysis pipeline comes with the advantage that our results can go through a complete replication analysis. Finally, the available pipeline could be used for future replication studies undergoing preregistration or published as registered report ( 90 ).…”

“Please Miss, Please!”: An observational study of young people’s social and emotional experiences post-transition to secondary school

“…All observations were conducted by the researcher, and thus the impact of researcher bias must be acknowledged. Literature expresses how the implicit beliefs of the researcher and expected outcomes of the study could unintentionally impact the participants’ behaviour and observations (Baldwin et al, 2020; Gao, 2020; Hamilton, 2020). Nevertheless, research into the influence of researcher bias implies ‘the pervasiveness of this influence among different areas of research is not known (and) how observer expectancies exert their influence is unclear’ (Kazdin, 2011, p. 88).…”

The cross‐over of statistical thinking and practices: A pandemic catalyst

“…Indeed, it is not unusual for researchers to iterate and request other data sets or other variables once limitations emerge on closer inspection when data access is granted. Baldwin et al 15 have proposed solutions including declaring prior access to data and the use of synthetic and hold-out samples. In this respect, prespecification has not advanced in the methodology guideline to stating the size of effect that would be regarded as meaningful.…”