Quantifying religiosity: a comparison of approaches based on categorical self-identification and multidimensional measures of religious activity

Subjective evidence evaluation survey for many-analysts studies

“…For instance, commentaries published on the Many-Analysts Religion Project (e.g. [ 42 , 49 ]) argued that project leaders should have reduced analytic variability arising from differences in the interpretation of the research question. For instance, instead of asking ‘Do religious people report higher well-being?’ it would be more beneficial to ask whether ‘ specific behaviours and/or beliefs benefit specific populations’ well-being or health in specific contexts’ [ 42 , p. 2].…”

“…[ 42 , 49 ]) argued that project leaders should have reduced analytic variability arising from differences in the interpretation of the research question. For instance, instead of asking ‘Do religious people report higher well-being?’ it would be more beneficial to ask whether ‘ specific behaviours and/or beliefs benefit specific populations’ well-being or health in specific contexts’ [ 42 , p. 2]. In a many-analysts project which aims to reduce analytic variability stemming from different interpretations of the research question as much as possible, capturing the subjective evaluations of the analysis teams may not be advised.…”

Subjective Evidence Evaluation Survey For Many-Analysts Studies

“…There might be situations, however, where manyanalysts projects essentially eliminate opportunity for analysts to choose which variables to assess. For instance, commentaries published on the Many-Analysts Religion Project (e.g., Edelsbrunner et al, 2023;Murphy & Martinez, 2023) argued that project leaders should have reduced analytic variability arising from differences in the interpretation of the research question. For instance, instead of asking "Do religious people report higher well-being?"…”

“…These analyses included structural equation modeling (McNamara, 2023), machine learning (van Lissa, 2023), and even multiverse analyses (Hanel & Zarzeczna, 2023;Krypotos et al, 2023). 1 Second, many teams presented more nuanced interpretations of the primary effect based on sub-group analyses or multivariate approaches (e.g., Atkinson et al, 2023;Murphy & Martinez, 2023;Pearson et al, 2023;Smith, 2023;Vogel et al, 2023) which helped determine the conditions under which the hypothesized relation occurred. Third, some teams raised concerns about measurement invariance in the data themselves (e.g., Ross et al, 2023;Schreiner et al, 2023).…”

“…Third, some teams raised concerns about measurement invariance in the data themselves (e.g., Ross et al, 2023;Schreiner et al, 2023). Others criticized the formulation of the research question (e.g., Edelsbrunner et al, 2023;Murphy & Martinez, 2023), an issue that surfaced in previous many-analysts projects (van Dongen et al, 2019). In sum, relying on a single reported effect from each team leaves no room for a more nuanced and detailed interpretation of the results and the underlying data (Hoogeveen, Sarafoglou, van Elk, et al, 2023).…”