Sample size and precision of estimates in studies of depression screening tool accuracy: A meta‐research review of studies published in 2018–2021

Abstract

Objectives Depression screening tool accuracy studies should be conducted with large enough sample sizes to generate precise accuracy estimates. We assessed the proportion of recently published depression screening tool diagnostic accuracy studies that reported sample size calculations; the proportion that provided confidence intervals (CIs); and precision, based on the width and lower bounds of 95% CIs for sensitivity and specificity. In addition, we assessed whether these results have improved since a previous review of studies published in 2013–2015. Methods MEDLINE was searched from January 1, 2018, through May 21, 2021. Results Twelve of 106 primary studies (11%) described a viable sample size calculation, which represented an improvement of 8% since the last review. Thirty‐six studies (34%) provided reasonably accurate CIs. Of 103 studies where 95% CIs were provided or could be calculated, seven (7%) had sensitivity CI widths of ≤10%, whereas 58 (56%) had widths of ≥21%. Eighty‐four studies (82%) had lower bounds of CIs <80% for sensitivity and 77 studies (75%) for specificity. These results were similar to those reported previously. Conclusion Few studies reported sample size calculations, and the number of included individuals in most studies was too small to generate reasonably precise accuracy estimates.

Transparency and completeness of reporting of depression screening tool accuracy studies: A meta‐research review of adherence to the Standards for Reporting of Diagnostic Accuracy Studies statement

“…However, we found that 72% of included studies did not specify the targeted sample size including how it was calculated. Similar trends were observed in reviews of depression screening tool accuracy studies published between (1) and 36 (34%) provided reasonably accurate confidence intervals (Nassar et al, 2022a). Importantly, accuracy studies with small samples sizes often fail to identify the most accurate cut-off and overstate accuracy estimates for the cut-offs they report (Bhandari et al, 2021).…”

“…However, we found that 72% of included studies did not specify the targeted sample size including how it was calculated. Similar trends were observed in reviews of depression screening tool accuracy studies published between (1) 2013 and 2015, where only three of 89 (3%) studies described a viable sample size calculation and 30 studies (34%) provided reasonably accurate confidence intervals around accuracy estimates (Thombs & Rice, 2016 ) and (2) 2018 and 2021, where only 12 of 106 (11%) studies described a viable sample size calculation and 36 (34%) provided reasonably accurate confidence intervals (Nassar et al., 2022a ). Importantly, accuracy studies with small samples sizes often fail to identify the most accurate cut‐off and overstate accuracy estimates for the cut‐offs they report (Bhandari et al., 2021 ).…”

“…This was part of a series of three meta‐research reviews that evaluated recently published studies of depression screening tool accuracy. The other two reviews examined the reporting of sample size calculations and precision of accuracy estimates (Nassar et al., 2022a ) and the characteristics of participants included in studies (Nassar et al., 2022b ). Prior to initiating the present study, a study protocol was posted on the Open Science Framework ( https://osf.io/5tvf3/ ).…”

‘Optimal’ cutoff selection in studies of depression screening tool accuracy using the PHQ‐9, EPDS, or HADS‐D: A meta‐research study

“…Any decision analysis to select cutoffs for use in future trials should rely upon evidence from large, high-quality meta-analyses. (Nassar et al, 2022). Furthermore, two-step methods, in which all participants with positive screens but only a proportion with negative screens are assessed with diagnostic interviews, could be used to obtain satisfactory precision for both sensitivity and specificity (Thombs et al, 2018).…”

“…If optimal cutoffs are identified, possible bias from data‐driven methods and imprecision of estimates should be noted, and these factors should be considered before author recommendations are made. As recommended by STARD, a priori sample size calculations should be conducted in depression screening tool accuracy studies to avoid drawing potentially misleading conclusions from overly small samples, but only approximately 10% of such studies report sample size calculations (Nassar et al., 2022). Furthermore, two‐step methods, in which all participants with positive screens but only a proportion with negative screens are assessed with diagnostic interviews, could be used to obtain satisfactory precision for both sensitivity and specificity (Thombs et al., 2018).…”