Accuracy of national key performance indicator reporting from two Aboriginal medical services: potential to underestimate the performance of primary health care

Abstract

The aim of the present study was to assess the accuracy of extracting national key performance indicator (nKPI) data for the Online Community Health Reporting Environment for Health Services (OCHREStreams) program using the Pen Computer Systems (Leichhardt, NSW, Australia) Clinical Audit Tool (CAT) from Communicare (Telstra Health Communicare Systems, Perth, WA, Australia), a commonly used patient information management system (PIMS) in Aboriginal primary care. Two Aboriginal Community-Controlled Health Services (ACCHSs) were recruited to the present study. A sample of regular clients aged ≥55 years from each ACCHS was selected and a subset of 13 nKPIs was examined. A manual case note audit of the nKPI subset within Communicare was undertaken by a clinician at each participating ACCHS and acted as a 'gold standard' comparator for three query methods: (1) internal Communicare nKPI reports; (2) PenCS CAT nKPI manual filtering (a third-party data-extraction tool); and (3) nKPI data submitted to the Improvement Foundation qiConnect portal. No errors were found in nKPI data extraction from Communicare using the CAT and subsequent submission to the qiConnect portal. However, the Communicare internal nKPI report included deceased clients and past patients, and we can be very confident that deceased clients and past patients are also included in the qiConnect portal data. This resulted in inflation of client denominators and an underestimation of health service performance, particularly for nKPIs recording activity in the past 6 months. Several minor errors were also detected in Communicare internal nKPI reports. CAT accurately extracts a subset of nKPI data from Communicare. However, given the widespread use of Communicare in ACCHSs, the inclusion of deceased clients and past patients in the OCHREStreams nKPI data program is likely to have resulted in systematic under-reporting of health service performance nationally. There has been limited validation of health data exported via data-extraction tools in Australia. More specifically, there are no current published data describing the accuracy of the CAT in mapping health data extracted from Communicare or the accuracy of internal nKPI reports generated by Communicare. Further, no systematic review has been undertaken to assess the accuracy of the nKPI data submission pathway from PIMSs at the health service level to the OCHREStreams qiConnect portal using the CAT. The CAT accurately extracts a subset of nKPI data from Communicare and accurately submits this to the qiConnect portal. Minor errors exist in some Communicare internal nKPI reports. The inclusion of deceased clients and past patients in the nKPI reporting system for ACCHSs is likely to have resulted in systematic under-reporting of health service performance nationally through this program. The inclusion of deceased clients and past patients in the OCHREStreams nKPI program limits the usefulness of these data for local quality improvement activities and national monitoring of health service performance for par...

Could health information systems enhance the quality of Aboriginal health promotion? A retrospective audit of Aboriginal health programs in the Northern Territory of Australia

“…As for information systems more generally [ 12 , 13 , 32 , 33 ], a significant constraint in realizing the potential of QIPPS was data quality. Information about some elements of health promotion, particularly related to project planning, were more readily available, and reliably collected, such as stating project goals and objectives, identifying the target group and health issues to be addressed.…”

“…EMRs are valuable sources of data that providers can use in making decisions about health care delivery. Indeed, health services have sought innovative ways to utilize these data to report on and improve the quality health care and health system performance for Aboriginal Australians [ 12 , 13 ]. EMRs and many other health information systems, however, are rarely designed or developed to capture, store or retrieve data about population-level services and activities.…”

Could health information systems enhance the quality of Aboriginal health promotion? A retrospective audit of Aboriginal health programs in the Northern Territory of Australia

“…As for information systems more generally (12,13,31,32), a signi cant constraint in realizing the potential of QIPPS was data quality. Information about some elements of health promotion, particularly related to project planning, were more readily available, and reliably collected, such as stating project goals and objectives, identifying the target group and health issues to be addressed.…”

Could health information systems enhance the quality of Aboriginal health promotion? A retrospective audit of Aboriginal health programs in the Northern Territory of Australia

“…EMRs are valuable sources of data that providers can use in making decisions about health care delivery. Indeed, health services have sought innovative ways to utilize these data to report on and improve the quality health care and health system performance for Aboriginal Australians (12,13). EMRs and many other health information systems, however, are rarely designed or developed to capture, store or retrieve data about populationlevel services and activities.…”