This article analyzes the range of system optimization activities taking place over an extended period following the implementation of computerized physician order entry and clinical decision support systems. We undertook 207 qualitative semi-structured interviews, 24 rounds of non-participant observations of meetings and system use, and collected 17 organizational documents in five hospitals over three time periods between 2011 and 2016. We developed a systematic analysis of system optimization activities with eight sub-categories grouped into three main categories. This delineates the range of system optimization activities including resolving misalignments between technology and clinical practices, enhancing the adopted system, and improving user capabilities to utilize/further optimize systems. This study highlights the optimization efforts by user organizations adopting multi-user, organization-spanning information technologies. Hospitals must continue to attend to change management for an extended period (up to 5 years post-implementation) and develop a strategy for long-term system optimization including sustained user engagement, training, and broader capability development to ensure smoother and quicker realization of benefits.

Background There is a need to identify approaches to reduce medication errors. Interest has converged on ePrescribing systems that incorporate computerised provider order entry and clinical decision support functionality. Objectives We sought to describe the procurement, implementation and adoption of basic and advanced ePrescribing systems; to estimate their effectiveness and cost-effectiveness; and to develop a toolkit for system integration into hospitals incorporating implications for practice from our research. Design We undertook a theoretically informed, mixed-methods, context-rich, naturalistic evaluation. Setting We undertook six longitudinal case studies in four hospitals (sites C, E, J and K) that did not have ePrescribing systems at the start of the programme (three of which went live and one that never went live) and two hospitals (sites A and D) with embedded systems. In the three hospitals that implemented systems, we conducted interviews pre implementation, shortly after roll-out and at 1 year post implementation. In the hospitals that had embedded systems, we conducted two rounds of interviews, 18 months apart. We undertook a three-round eDelphi exercise involving 20 experts to identify 80 clinically important prescribing errors, which were developed into the Investigate Medication Prescribing Accuracy for Critical error Types (IMPACT) tool. We elicited the cost of an ePrescribing system at one (non-study) site and compared this with the calculated ‘headroom’ (the upper limit that the decision-maker should pay) for the systems (sites J, K and S) for which effectiveness estimates were available. We organised four national conferences and five expert round-table discussions to contextualise and disseminate our findings. Intervention The implementation of ePrescribing systems with either computerised provider order entry or clinical decision support functionality. Main outcome measures Error rates were calculated using the IMPACT tool, with changes over time represented as ratios of error rates (as a proportion of opportunities for errors) using Poisson regression analyses. Results We conducted 242 interviews and 32.5 hours of observations and collected 55 documents across six case studies. Implementation was difficult, particularly in relation to integration and interfacing between systems. Much of the clinical decision support functionality in embedded sites remained switched off because of concerns about over alerting. Getting systems operational meant that little attention was devoted to system optimisation or secondary uses of data. The prescriptions of 1244 patients were audited pre computerised provider order entry and 1178 post computerised provider order entry implementation of system A at sites J and K, and system B at site S. A total of 21,138 opportunities for error were identified from 28,526 prescriptions. Across the three sites, for those prescriptions for which opportunities for error were identified, the error rate was found to reduce significantly post computerised provider order entry implementation, from 5.0% to 4.0% (p < 0.001). Post implementation, the overall proportion of errors (per opportunity) decreased significantly in sites J and S, but remained similar in site K, as follows: 4.3% to 2.8%, 7.4% to 4.4% and 4.0% to 4.4%, respectively. Clinical decision support implementation by error type was found to differ significantly between sites, ranging from 0% to 88% across clinical contraindication, dose/frequency, drug interactions and other error types (p < 0.001). Overall, 43 out of 78 (55%) of the errors had some degree of clinical decision support implemented in at least one of the hospitals. For the site in which no improvement was detected in prescribing errors (i.e. site K), the ePrescribing system represented a cost to the service for no countervailing benefit. Cost-effectiveness rose in proportion to reductions in error rates observed in the other sites (i.e. sites J and S). When a threshold value of £20,000 was used to define the opportunity cost, the system would need to cost less than £4.31 per patient per year, even in site S, where effectiveness was greatest. We produced an ePrescribing toolkit (now recommended for use by NHS England) that spans the ePrescribing life cycle from conception to system optimisation. Limitations Implementation delays meant that we were unable to employ the planned stepped-wedge design and that the assessment of longer-term consequences of ePrescribing systems was impaired. We planned to identify the complexity of ePrescribing implementation in a number of contrasting environments, but the small number of sites means that we have to infer findings from this programme with considerable care. The lack of transparency regarding system costs is a limitation of our method. As with all health economic analyses, our analysis is subject to modelling assumptions. The research was undertaken in a modest number of early adopters, concentrated on high-risk prescribing errors and may not be generalisable to other hospitals. Conclusions The implementation of ePrescribing systems was challenging. However, when fully implemented the ePrescribing systems were associated with a reduction in clinically important prescribing errors and our model suggests that such an effect is likely to be more cost-effective when clinical decision support is available. Careful system configuration considering clinical processes and workflows is important to achieving these potential benefits and, therefore, our findings may not be generalisable to all system implementations. Future work Formative and summative evaluations of efforts will be central to promote learning across settings. Other priorities emerging from this work include the possibility of learning from international experiences and the commercial sector. Funding This project was funded by the National Institute for Health and Care Research (NIHR) Programme Grants for Applied Research programme and will be published in full in Programme Grants for Applied Research; Vol. 10, No. 7. See the NIHR Journals Library website for further project information.