Effects and costs of implementing predictive risk stratification in primary care: a randomised stepped wedge trial

Abstract

AimWe evaluated the introduction of a predictive risk stratification model (PRISM) into primary care. Contemporaneously National Health Service (NHS) Wales introduced Quality and Outcomes Framework payments to general practices to focus care on those at highest risk of emergency admission to hospital. The aim of this study was to evaluate the costs and effects of introducing PRISM into primary care.MethodsRandomised stepped wedge trial with 32 general practices in one Welsh health board. The intervention comprised: PRISM software; practice-based training; clinical support through two ‘general practitioner (GP) champions’ and technical support. The primary outcome was emergency hospital admissions.ResultsAcross 230 099 participants, PRISM implementation increased use of health services: emergency hospital admission rates by 1 % when untransformed (while change in log-transformed rate ΔL=0.011, 95% CI 0.010 to 0.013); emergency department (ED) attendance rates by untransformed 3 % (while ΔL=0.030, 95% CI 0.028 to 0.032); outpatient visit rates by untransformed 5 % (while ΔL=0.055, 95% CI 0.051 to 0.058); the proportion of days with recorded GP activity by untransformed 1 % (while ΔL=0.011, 95% CI 0.007 to 0.014) and time in hospital by untransformed 3 % (while ΔL=0.029, 95% CI 0.026 to 0.031). Thus NHS costs per participant increased by £76 (95% CI £46 to £106).ConclusionsIntroduction of PRISM resulted in a statistically significant increase in emergency hospital admissions and use of other NHS services without evidence of benefits to patients or the NHS.

Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness

“…The vast majority of published clinical prediction models are never used in clinical practice 19. One reason for this is the lack of a specific clinical decision making process that the model could meaningfully inform or optimise; simply predicting future events on their own might not help a clinician do anything differently20 (in other words: just because we can, it does not mean we should). This is an important departure from the lone wolf attitude, which has helped foster innovation over the past few decades in ML/AI for health.…”

Emergency admission risk stratification tools in UK primary care: a cross-sectional survey of availability and use

“…32 The findings justified the cautious approach, as the intervention was shown to be ineffective and resulted in increased costs, emergency admissions, and bed days. 13,32 Further studies published after the widespread roll out of EARS, including a systematic review of case management for at-risk individuals (not necessarily identified through EARS tools), and an observational study of multidisciplinary case management for risk stratified patients, found increased costs and health service use. 15,33 It is far from uncommon for health innovations to be introduced at scale and great cost without an understanding of their costs and effects.…”

“…Of the studies that use EARS to select patients for community interventions (typically case management or virtual wards), none reported reductions in admissions, cost savings, or significant patient benefit. [13][14][15][16] Although there is no strong evidence supporting the clinical or cost effectiveness of interventions incorporating EARS tools, there has been concerted political and practical investment in EARS across the developed world. [17][18][19] In the UK, state funding supported the development of risk models: Patients at Risk of Readmission (PARR) and the Combined Model in England; Scottish Patients at Risk of Readmission and Admission (SPARRA) in Scotland; and Predictive Risk Stratification Model (PRISM) in Wales.…”

Quality & safety in the literature: May 2020

“…For example, in 2018, Persell and colleagues4 noted worse blood pressure at 12 months in a group of patients assigned to an electronic health record-based medication management tool compared with the control. Readers of BMJ Quality & Safety may also recall a 2019 study in which utilisation of a model to estimate risk of ED presentation in 32 general practices in Wales resulted in increased ED utilisation 5. The accompanying editorial reflected on a number of examples of unintended consequences of quality improvement efforts 6.…”