Supporting the implementation of new healthcare technologies by investigating generalisability of pilot studies using area-level statistics

Abstract

Background Implementation of new technologies into national health care systems requires careful capacity planning. This is sometimes informed by data from pilot studies that implement the technology on a small scale in selected areas. A critical consideration when using implementation pilot studies for capacity planning in the wider system is generalisability. We studied the feasibility of using publicly available national statistics to determine the degree to which results from a pilot might generalise for non-pilot areas, using the English human papillomavirus (HPV) cervical screening pilot as an exemplar. Methods From a publicly available source on population indicators in England (“Public Health Profiles”), we selected seven area-level indicators associated with cervical cancer incidence, to produce a framework for post-hoc pilot generalisability analysis. We supplemented these data by those from publicly available English Office for National Statistics modules. We compared pilot to non-pilot areas, and pilot regimens (pilot areas using the previous standard of care (cytology) vs. the new screening test (HPV)). For typical process indicators that inform real-world capacity planning in cancer screening, we used standardisation to re-weight the values directly observed in the pilot, to better reflect the wider population. A non-parametric quantile bootstrap was used to calculate 95% confidence intervals (CI) for differences in area-weighted means for indicators. Results The range of area-level statistics in pilot areas covered most of the spectrum observed in the wider population. Pilot areas were on average more deprived than non-pilot areas (average index of multiple deprivation 24.8 vs. 21.3; difference: 3.4, 95% CI: 0.2–6.6). Participants in HPV pilot areas were less deprived than those in cytology pilot areas, matching area-level statistics. Differences in average values of the other six indicators were less pronounced. The observed screening process indicators showed minimal change after standardisation for deprivation. Conclusions National statistical sources can be helpful in establishing the degree to which the types of areas outside pilot studies are represented, and the extent to which they match selected characteristics of the rest of the health care system ex-post. Our analysis lends support to extrapolation of process indicators from the HPV screening pilot across England.

Rede nacional de testes moleculares para detecção de Chlamydia trachomatis e Neisseria gonorrhoeae: experiência de implantação-piloto no Brasil

“…A implantação de uma nova tecnologia na rede de saúde partindo de um estudo-piloto é uma prática comum na área de IST no Brasil 12 , 13 , 14 , sendo também reconhecida mundialmente como uma iniciativa eficaz para avaliar o desempenho dessa nova tecnologia na população local, identificar as fortalezas do processo e conhecer as barreiras que necessitam ser superadas, visando o sucesso da implantação definitiva 15 , 16 , 17 , 18 .…”

Dynamic data-enabled stratified sampling for trial invitations with application in NHS-Galleri

“…Subsequent largerscale pilots and analyses are usually needed to evaluate and help plan implementation. 23 In conclusion, healthy volunteer effects and adequate representation have been identified as a problem for many years 24 but arguably little progress has been made in reducing the impact even with judicious recruitment strategies. We hope that our data-driven stratified sampling methodology might be applied elsewhere to enable future studies to better represent their target population, improve equity, diversity and inclusion of trial participants, and account for healthy volunteer effects.…”

“…Subsequent larger-scale pilots and analyses are usually needed to evaluate and help plan implementation. 23 …”