Digital methods to enhance the usefulness of patient experience data in services for long-term conditions: the DEPEND mixed-methods study

Abstract

<jats:sec id="abs1-1"> <jats:title>Background</jats:title> <jats:p>Collecting NHS patient experience data is critical to ensure the delivery of high-quality services. Data are obtained from multiple sources, including service-specific surveys and widely used generic surveys. There are concerns about the timeliness of feedback, that some groups of patients and carers do not give feedback and that free-text feedback may be useful but is difficult to analyse.</jats:p> </jats:sec> <jats:sec id="abs1-2"> <jats:title>Objective</jats:title> <jats:p>To understand how to improve the collection and usefulness of patient experience data in services for people with long-term conditions using digital data capture and improved analysis of comments.</jats:p> </jats:sec> <jats:sec id="abs1-3"> <jats:title>Design</jats:title> <jats:p>The DEPEND study is a mixed-methods study with four parts: qualitative research to explore the perspectives of patients, carers and staff; use of computer science text-analytics methods to analyse comments; co-design of new tools to improve data collection and usefulness; and implementation and process evaluation to assess use of the tools and any impacts.</jats:p> </jats:sec> <jats:sec id="abs1-4"> <jats:title>Setting</jats:title> <jats:p>Services for people with severe mental illness and musculoskeletal conditions at four sites as exemplars to reflect both mental health and physical long-terms conditions: an acute trust (site A), a mental health trust (site B) and two general practices (sites C1 and C2).</jats:p> </jats:sec> <jats:sec id="abs1-5"> <jats:title>Participants</jats:title> <jats:p>A total of 100 staff members with diverse roles in patient experience management, clinical practice and information technology; 59 patients and 21 carers participated in the qualitative research components.</jats:p> </jats:sec> <jats:sec id="abs1-6"> <jats:title>Interventions</jats:title> <jats:p>The tools comprised a digital survey completed using a tablet device (kiosk) or a pen and paper/online version; guidance and information for patients, carers and staff; text-mining programs; reporting templates; and a process for eliciting and recording verbal feedback in community mental health services.</jats:p> </jats:sec> <jats:sec id="abs1-7"> <jats:title>Results</jats:title> <jats:p>We found a lack of understanding and experience of the process of giving feedback. People wanted more meaningful and informal feedback to suit local contexts. Text mining enabled systematic analysis, although challenges remained, and qualitative analysis provided additional insights. All sites managed to collect feedback digitally; however, there was a perceived need for additional resources, and engagement varied. Observation indicated that patients were apprehensive about using kiosks but often would participate with support. The process for collecting and recording verbal feedback in mental health services made sense to participants, but was not successfully adopted, with staff workload and technical problems often highlighted as barriers. Staff thought that new methods were insightful, but observation did not reveal changes in services during the testing period.</jats:p> </jats:sec> <jats:sec id="abs1-8"> <jats:title>Conclusions</jats:title> <jats:p>The use of digital methods can produce some improvements in the collection and usefulness of feedback. Context and flexibility are important, and digital methods need to be complemented with alternative methods. Text mining can provide useful analysis for reporting on large data sets within large organisations, but qualitative analysis may be more useful for small data sets and in small organisations.</jats:p> </jats:sec> <jats:sec id="abs1-9"> <jats:title>Limitations</jats:title> <jats:p>New practices need time and support to be adopted and this study had limited resources and a limited testing time.</jats:p> </jats:sec> <jats:sec id="abs1-10"> <jats:title>Future work</jats:title> <jats:p>Further research is needed to improve text-analysis methods for routine use in services and to evaluate the impact of methods (digital and non-digital) on service improvement in varied contexts and among diverse patients and carers.</jats:p> </jats:sec> <jats:sec id="abs1-11"> <jats:title>Funding</jats:title> <jats:p>This project was funded by the NIHR Health Services and Delivery Research programme and will be published in full in <jats:italic>Health Services and Delivery Research</jats:italic>; Vol. 8, No. 28. See the NIHR Journals Library website for further project information.</jats:p> </jats:sec>

Co-designing New Tools for Collecting, Analysing and Presenting Patient Experience Data in NHS Services: Working in Partnership With Patients and Carers

“…The DEPEND study involved staff, patients and carers research participants from multiple sites co-producing the research with a core PPIE component embedded within each of the work packages (32). We were able to adopt a shared approach to co-design to shape a suite of new tools to test the usefulness and relevance of patient experience data.…”

“…In this paper, we focus on the PPIE and co-design components of a project entitled 'Developing and Enhancing the Usefulness of Patient Experience and Narrative Data' (the DEPEND study: (32)). While a dedicated PPIE group was active throughout the mixedmethods study, it was particularly important in the co-design of the new tools and associated guidance (the 'toolkit') and their evaluation.…”

“…Exploring alternative methods, such as adopting a machine learning classi cation approach for digitally collected free patient comments, has been reported in the literature (45)(46)(47). The DEPEND study builds on the aforementioned work in a number of ways and this is reported elsewhere (32,48,49). The aim of the involvement component to the DEPEND study was to inform the research process within each of the four study work packages to ensure integration of involvement into the intervention development process from the start.…”

“…Using the new methods for data visualisation, the extracted data was presented to participants to facilitate integration and contrast with other sources. The total number of people who participated in the three data collection phases for DEPEND is reported within the nal report (32). Note: number of participants that had taken part previously is in brackets; shaded area indicates data from patients in sites C1 and C2 were not collected for the co-design phase.…”

“…Our recommendation that information and bespoke guidance should be provided ready for the set-up of the new tools was to be monitored and adapted to incorporate the feedback data throughout the testing period (32,48,49). As part of the later phase of the co-design and implementation phase, visual feedback reports that included summaries of quantitative data and free text comments were split by sentiment (positive and negative comments: (32)).…”