Sample size requirements are not being considered in studies developing prediction models for binary outcomes: a systematic review

Abstract

Background Having an appropriate sample size is important when developing a clinical prediction model. We aimed to review how sample size is considered in studies developing a prediction model for a binary outcome. Methods We searched PubMed for studies published between 01/07/2020 and 30/07/2020 and reviewed the sample size calculations used to develop the prediction models. Using the available information, we calculated the minimum sample size that would be needed to estimate overall risk and minimise overfitting in each study and summarised the difference between the calculated and used sample size. Results A total of 119 studies were included, of which nine studies provided sample size justification (8%). The recommended minimum sample size could be calculated for 94 studies: 73% (95% CI: 63–82%) used sample sizes lower than required to estimate overall risk and minimise overfitting including 26% studies that used sample sizes lower than required to estimate overall risk only. A similar number of studies did not meet the ≥ 10EPV criteria (75%, 95% CI: 66–84%). The median deficit of the number of events used to develop a model was 75 [IQR: 234 lower to 7 higher]) which reduced to 63 if the total available data (before any data splitting) was used [IQR:225 lower to 7 higher]. Studies that met the minimum required sample size had a median c-statistic of 0.84 (IQR:0.80 to 0.9) and studies where the minimum sample size was not met had a median c-statistic of 0.83 (IQR: 0.75 to 0.9). Studies that met the ≥ 10 EPP criteria had a median c-statistic of 0.80 (IQR: 0.73 to 0.84). Conclusions Prediction models are often developed with no sample size calculation, as a consequence many are too small to precisely estimate the overall risk. We encourage researchers to justify, perform and report sample size calculations when developing a prediction model.

Prognostic Models Predicting Clinical Outcomes in Patients Diagnosed with Visceral Leishmaniasis: A Systematic Review

“…Model overfitting, a common source of bias, occurs when the modelling process captures idiosyncratic random variation in the development dataset that is not reflective of the true patient population. Small sample sizes, grouping of continuous predictors, and univariable predictor selection can all introduce bias through overfitting, leading to optimistic performance estimates and exaggerated risk predictions (7,25,44).…”

CT-based pancreatic radiomics predicts secondary loss of response to infliximab in biologically naïve patients with Crohn’s disease

“…In view of this, a post hoc analysis of sample size is performed, which finds that although the current sample size does not fully meet the criteria proposed by Riley et al ( n = 289; events per predictor parameter [EPP]: 14.45) [ 54 ], the EPP of the current model reaches 12.512. Furthermore, instability plot based on bootstrap model ( b = 500 times) demonstrates that the existing model is relatively stable (mean absolute percentage error: 0.0585) [ 55 , 56 ]. Additionally, the definition of the outcome in this study is not sufficiently objective.…”

Machine Learning Meets Psychological Data: Challenges and Future Directions

“…Instead, the adequacy of the sample size is usually implicitly evaluated in relation to the model performance. However, recent research has demonstrated that many datasets used to build and evaluate prediction models have been undersized, increasing the risk of bias (Wynants et al, 2020), and potentially giving rise to inaccurate conclusions (Dhiman et al, 2023). Thus, more focus could go towards implementing standardized practices (e.g., as in medicine, see Riley et al (2019aRiley et al ( ,b, 2020) for data in psychology (i.e., survey data, which likely has measurement error) to ensure adequate sample sizes for machine learning analyses.…”