Estimating the replicability of technology education research

Buckley, Jeffrey; Hyland, Tomás; Seery, Niall

doi:10.1007/s10798-022-09787-6

Cited by 4 publications

(2 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the general estimate for social psychology (Bartoš & Schimmack, 2022) or construal level theory (Maier et al, 2022) seems to be slightly better. Furthermore, top medical journals (Schimmack & Bartoš, n.d.), technology education research (Buckley et al, 2022), and organizational research (Gupta & Bosco, 2023), or tools and interventions for mitigating risks for gambling harm (McAuliffe et al, 2021), forced confabulation effect (Riesthuis et al, 2023), and social media use and self-esteem van Anen, 2022 seem to be of much higher quality. Finally, see Replicability-Index Sensitivity analysis to the hand-coding of the results shows robustness of results.…”

Section: Discussionmentioning

confidence: 99%

Untrustworthy Evidence in Dishonesty Research

Bartoš

2024

View full text Add to dashboard Cite

Replicable and reliable research is essential for cumulative science and its applications in practice. This article examines the quality of research on dishonesty using a sample of 286 hand-coded test statistics from 99 articles. Z-curve analysis indicates a low expected replication rate, a high proportion of missing studies, and an inflated false discovery risk. Test of insufficient variance (TIVA) finds that 11/61 articles with multiple test statistics contain results that are ``too-good-to-be-true''. Sensitivity analysis confirms the robustness of the findings. In conclusion, caution is advised when relying on or applying the existing literature on dishonesty.

show abstract

Section: Discussionmentioning

confidence: 99%

Untrustworthy Evidence in Dishonesty Research

Bartoš

2024

View full text Add to dashboard Cite

show abstract

“…However, the general estimate for social psychology (Bartoš & Schimmack, 2022) or construal level theory (Maier et al, 2022) seems to be slightly better. Furthermore, top medical journals (Schimmack & Bartoš, 2023), technology education research (Buckley et al, 2022), and organizational research (Gupta & Bosco, 2023), or tools and interventions for mitigating risks for gambling harm (McAuliffe et al, 2021), forced confabulation effect (Riesthuis et al, 2023), and social media use and self-esteem van Anen (2022) seem to be of much higher quality. Finally, see Replicability-Index blog posts for psychological journals' specific estimates based on automatically extracted test statistics (e.g., https://replicationindex.com/2022/01/26/rr21/ for 2022 ratings).…”

Section: Discussionmentioning

confidence: 99%

The Untrustworthy Evidence in Dishonesty Research

Bartoš¹

2023

Preprint

View full text Add to dashboard Cite

Replicable and reliable research is essential for cumulative science and its applications in practice. This article examines the quality of research on dishonesty using a sample of 278 hand-coded test statistics from 99 articles. Z-curve analysis indicates a low expected replication rate, a high proportion of missing studies, and an inflated false discovery risk. Test of insufficient variance (TIVA) finds that 11/61 articles with multiple test statistics contain results that are `too-good-to-be-true'. Sensitivity analysis confirms the robustness of the findings. In conclusion, caution is advised when relying on or applying the existing literature on dishonesty.

show abstract