The ALFA (Activity Log Files Aggregation) Toolkit: A Method for Precise Observation of the Consultation

Abstract

BackgroundThere is a lack of tools to evaluate and compare Electronic patient record (EPR) systems to inform a rational choice or development agenda.ObjectiveTo develop a tool kit to measure the impact of different EPR system features on the consultation.MethodsWe first developed a specification to overcome the limitations of existing methods. We divided this into work packages: (1) developing a method to display multichannel video of the consultation; (2) code and measure activities, including computer use and verbal interactions; (3) automate the capture of nonverbal interactions; (4) aggregate multiple observations into a single navigable output; and (5) produce an output interpretable by software developers. We piloted this method by filming live consultations (n = 22) by 4 general practitioners (GPs) using different EPR systems. We compared the time taken and variations during coded data entry, prescribing, and blood pressure (BP) recording. We used nonparametric tests to make statistical comparisons. We contrasted methods of BP recording using Unified Modeling Language (UML) sequence diagrams.ResultsWe found that 4 channels of video were optimal. We identified an existing application for manual coding of video output. We developed in-house tools for capturing use of keyboard and mouse and to time stamp speech. The transcript is then typed within this time stamp. Although we managed to capture body language using pattern recognition software, we were unable to use this data quantitatively. We loaded these observational outputs into our aggregation tool, which allows simultaneous navigation and viewing of multiple files. This also creates a single exportable file in XML format, which we used to develop UML sequence diagrams. In our pilot, the GP using the EMIS LV (Egton Medical Information Systems Limited, Leeds, UK) system took the longest time to code data (mean 11.5 s, 95% CI 8.7-14.2). Nonparametric comparison of EMIS LV with the other systems showed a significant difference, with EMIS PCS (Egton Medical Information Systems Limited, Leeds, UK) (P = .007), iSoft Synergy (iSOFT, Banbury, UK) (P = .014), and INPS Vision (INPS, London, UK) (P = .006) facilitating faster coding. In contrast, prescribing was fastest with EMIS LV (mean 23.7 s, 95% CI 20.5-26.8), but nonparametric comparison showed no statistically significant difference. UML sequence diagrams showed that the simplest BP recording interface was not the easiest to use, as users spent longer navigating or looking up previous blood pressures separately. Complex interfaces with free-text boxes left clinicians unsure of what to add.ConclusionsThe ALFA method allows the precise observation of the clinical consultation. It enables rigorous comparison of core elements of EPR systems. Pilot data suggests its capacity to demonstrate differences between systems. Its outputs could provide the evidence base for making more objective choices between systems.

Investigating the contribution of physician assistants to primary care in England: a mixed-methods study

“…It was initially developed in the 1990s and demonstrated to have validity [108][109][110] and reliability across levels of clinical competence. 107 Testing of reliability found a range of mean scores between 51.3% and 70.2% in one study, depending on levels of experience, 110 and between 45% and 75% in another looking at the effect of interruption. 109 Although it has been criticised, no better package has been developed.…”

“…The recordings were made using ALFA: a computer-mediated consultation observation technique developed to provide an analysable overview of the consultation. 107 Each consultation was assessed by two GPs independently using the LAP 105 for assessing competence in general practice on dimensions of interviewing and history taking, patient management, problem solving and behaviour and relationship with patients.…”

Clinicians were oblivious to incorrect logging of test dates and the associated risks in an online pathology application: a case study

“…Video is acceptable and a useful media to observe workflow and can be combined with other data gathering approaches. 26 Video studies could be used to capture current ways of working and the impact and risk associated with the introduction of new technologies on it. 27 …”

Are we setting about improving the safety of computerised prescribing in the right way? A workshop report

“…In vivo observation of real consultations is needed, perhaps starting with simple counts of the number of times that the alerts are overridden, possibly followed by use of techniques such as the multichannel video to provide a greater understanding of where these alerts improve safety. 20 Software designers should continue to work closely with users and modify software to suit the practical requirements of healthcare professionals during the clinical consultation. The 'costs' of the prescribing alerts, including the effect on the clinician's cognitive performance, should be taken into account during the design process.…”