Inter-rater Reliability of a Classification System for Hospital Adverse Drug Event Reports

Abstract

Hospital pharmacovigilance systems frequently classify adverse drug event (ADE) reports on various axes such as severity and type of outcome in an attempt to better detect changes in the frequency of certain types of ADEs. The aim of this study was to measure the inter-observer reliability of an ADE classification system. Two pharmacists and two internal medicine physicians reviewed 150 pharmacist-generated ADE reports and used a structured form to classify reports on four domains: the presence or absence of process measures leading to ADE; the individual who initiated the process that potentially leads to ADE; the severity of ADE; and whether the ADE was related to dose. There was wide variation in inter-observer reliability of different elements in a classification system for ADEs. Agreement on specific processes associated with ADEs ranged from poor to moderate, which limits the ability to target accurately processes to improve drug utilization.

Reliability of the assessment of preventable adverse drug events in daily clinical practice

“…In the study by Snyder et al 16 each case was assessed, then classified after discussion by the individual raters before moving to the next case. This is in contrast to both the study of Haynes et al 15 (in which raters individually classified only without discussion) and our study, in which only after all cases were classified individually, consensus was reached in a subsequent meeting. We chose for this method, because our goal was to determine agreement between individual assessors prior to consensus building in order to evaluate the 'average' healthcare professional opinion on preventable ADEs.…”

“…[12][13][14] Our findings underline it is the same for assessing preventable ADEs and their severity from medical charts in everyday practice. This is in line with the recently published study of Haynes et al 15 but surprisingly is in contrast with high inter-rater agreement found by Forrey et al 9 and Snyder et al 16 So, why do some studies find such poor agreement while others do not? First, the level of detail of case information given to the assessors differs.…”

“…This study shows that there is only fair agreement on the assessment of an adverse event being actually preventable harm. Our conclusion is the same as Haynes et al, 15 it is still a challenge to assess ADEs in a reliable way. As long as the reliability is low, it will be difficult to determine the absolute number of preventable ADEs.…”

Standardising the Classification of Harm Associated with Medication Errors: The Harm Associated with Medication Error Classification (HAMEC)

“…Unfortunately, poor record quality is consistently cited to be a significant limitation in assessing adverse drug events [26–38]. The difficulty of classifying either potential or actual harm is reflected in reports of low inter-rater reliability among clinicians [39–41], although experience as a clinician, and engaging more than two clinicians has been shown to improve the consistency of these ratings [42].…”