Validating Syndromic Data for Opioid Overdose Surveillance in Florida

Abstract

ObjectiveAssess the validity of Florida (FL) Enhanced State Opioid Overdose Surveillance (ESOOS) non-fatal syndromic case definitions.IntroductionIn 2017, FL Department of Health (DOH) became one of thirty-two states plus Washington, D.C funded by the Center for Disease Control and Prevention (CDC) under the ESOOS program. One of the objectives of this funding was to increase the timeliness of reporting on non-fatal opioid overdoses through syndromic surveillance utilizing either the emergency department (ED) or Emergency Medical Services (EMS) data systems. Syndromic case validation is an essential requirement under ESOOS for non-fatal opioid-involved overdose (OIOD). FL’s ESOOS program conducted OIOD validation and quality monitoring of EMS case definitions, using data from FL’s Emergency Medical Services Tracking and Reporting System (EMSTARS). We examined measurement validity with OIOD cases identified from FL’s statewide hospital billing database, FL Agency for Health Care Administration (AHCA).MethodsFrom FL-EMSTARS, we extracted EMS data where the type of service requested was a 911 response, the patient was treated then transported by EMS to a hospital facility in Florida and was 11 years of age or older. Additionally, all incident-patient encounters excluded those who were dead at the scene. We included all responses with dispatch dates between January 1, 2016, and December 31, 2016. From FL-AHCA, we extracted ED and inpatient discharge information with admission dates and patient age covering the same ranges as our EMS encounters. We classified FL-EMSTARS cases based on combinations, like that of Rhode Island,1 using providers primary impression (PPI), providers secondary impression (PSI) and response to the administration of naloxone. FL-AHCA cases were defined by the following T and F codes from the International Classification of Diseases 10: T40.0-T40.4, T40.60, T40.69, F11.12, F11.120, F11.121, F11.122, F11.129, F11.22, F11.220, F11.221, F11.222, F11.229, F11.92, F11.920, F11.921, F11.922, F11.929. For all “T” codes, the 6th character was either a “1” or “4,” because ESOOS is focused on unintentional and undetermined drug overdoses, ergo we excluded ED visits that are related to intentional self-harm (i.e., “2”) or assault (i.e., “3”). Lastly, for all “T” codes, the 7th character we included was the initial ED encounter (i.e., “A”) because the purpose of the system is to capture increases or decreases in acute overdoses. To improve our match rate, account for typographical errors, and account for the discriminatory power some values may contain, we employed probabilistic linkage using Link Plus software developed by the CDC Cancer Division. Blocking occurred among social security number (SSN), event date, patient age in years, and date of birth (DOB). Next, we matched both datasets on ten variables: event date, age, sex, DOB, ethnicity, facility code, hospital zip code, race, SSN, and patient’s residence zip code. Further pruning was performed to ensure all matches were within a 24-hour time interval. Data management and statistical analyses were performed using SAS® statistical software, version 9.4 (SAS Institute Inc., Cary, NC, USA). We assessed EMS measurement validity by sensitivity, specificity, and positive predictive value (PPV). Next, risk factors were identified by stepwise multivariable logistic regression to improve the accuracy of the FL-ESOOS definition. Significant risk factors from the parsimonious multivariable model were used to simulate unique combinations to estimate the maximum sensitivity and PPV for OIOD.ResultsPrior to merging, FL-EMSTARS contained 1,308,825 unique incident-patient records, where FL-AHCA contained 8,862,566 unique incident-patient records. Of these, we conservatively linked 892,593 (68.2%) of the FL-EMSTARS dataset with FL-AHCA. Our probabilistic linkage represents an 18.2% linkage improvement over previous FL-DOH deterministic strategies (J Jiang, unpublished CSTE presentation, 2018). Among the matched pairs we estimated 8,526 OIOD, 0.96% prevalence, using the FL-AHCA case definition. Whereas the FL-ESOOS syndromic case definition estimated 6,188 OIOD, 0.69% prevalence. The FL-ESOOS OIOD syndromic case definition demonstrated 31.64% sensitivity, 99.61% specificity, and 43.60% PPV. Among false negatives, the response to administrated naloxone among OIOD was 39.37% “not known,” 37.95% “unchanged,” and 0.28% “worse.” We altered the FL-ESOOS EMSTARS case definition for OIOD to include those who were administered naloxone regardless of their response to the medication. We observed 12.37% sensitivity increase to 44.01%, 0.56% specificity decrease to 99.05%, and 12.78% PPV decrease to 30.82%.Are final multivariable model is as follows: lnOdds(Opioid Overdose)= 12.66 – 0.5459(Med Albuterol) – 0.9568(Med Aspirin) – 0.5765(Med Midazolam Hydrochloride) – 0.8690(Med Morphine Sulfate) + 1.4103 (Med Naloxone) – 0.7694(Med Nitroglycerine) + 0.3622(Med Oxygen) – 0.3702(Med Phenergan) – 0.8820(Med Epinephrine 1:10000) – 0.7397(Med Fentanyl) – 0.6376(Med Sodium Bicarbonate) – 0.2725(Med Normal Saline) + 0.3935(Med Other-Not Listed) + 0.6300(PPI General Malaise) + 0.8476(PPI Other, Non-Traumatic Pain) + 0.8725(PPI Airway Obstruction) + 0.4808(PPI Allergic reaction) + 1.4948(PPI Altered level of consciousness) + 1.5481(PPI Behavioral/psychiatric disorder) + 1.3843(PPI Cardiac arrest) + 2.3913(PPI Poisoning/drug ingestion) + 2.2418(PPI Intentional Drug Use; Related Problems) + 0.2783(PPI Respiratory distress) + 2.0305(PPI Respiratory arrest) + 0.4292(PPI Stroke/CVA) + 0.5402(PPI Syncope/fainting) + 0.5219(PSI Other, Non-Traumatic Pain) + 0.9355(PSI Allergic reaction) + 0.3521(PSI Altered level of consciousness) + 0.9036(PSI Poisoning/drug ingestion) + 0.9661(PSI Intentional Drug Use; Related Problems) + 0.3766(PSI Respiratory Distress) + 1.1802(PSI Respiratory Arrest).We plotted the multivariable sensitivity and PPV by probaiblity cutoff value to determine which would produce the best discrimination (see Figure 1). By incorporating a probability cutoff value ≥ 0.22, we can inprove both sensitivity and PPV. Specifically, we can achieve 45.48% sensitivity, 99.32% specificity, and 45.48% PPV.ConclusionsThe sensitivity of the FL-ESOOS surveillance system is not generally high but could still be useful if subsequent validation shows sensitivity stability. Regarding maximizing FL-ESOOS sensitivity and PPV, we deomonstrated that our mulitvariable model with an appropriate probability cutoff value performes better than the current case definition. This study contributes to the limited literature on Florida non-fatal opioid overdoses with a specific emphasis on validating EMS records. New unique indicator combinations are possible to increase sensitivity and PPV but should be thoroughly investigated to balance the tradeoffs to optimize the system’s ability to detect non-fatal overdoses and to discriminate true cases.References1. Rhode Island Department of Health. Rhode Island Enhanced State Opioid Overdose Surveillance (ESOOS) Case Definition For Emergency Medical Services (EMS).; 2017.2. Jiang J, Mai A, Card K, Sturms J, McCoy S. EMS Naloxone Administration for Implication of Opioid Overdose. Presentation presented at the: 2018; CSTE Annual Conference.

Using Emergency Medical Services Data to Monitor Nonfatal Opioid Overdoses in Real Time

“…18 When compared with the modeled definition, our sensitivity was substantially higher (90% vs 45%) in identifying opioid overdose-related EMS runs. 17 Although the natural language processing algorithm correctly identified 98.6% of opioid misuse cases, because of the differences in our definitions (opioid overdose vs opioid misuse), it is difficult to compare results. 18 One strength of this work is the generalizability of the case definition.…”

“…Since we developed the case definition to identify nonfatal opioid overdose-related EMS runs, other groups have also developed opioid case definitions using multivariable logistic regression 17 and natural language processing. 18 When compared with the modeled definition, our sensitivity was substantially higher (90% vs 45%) in identifying opioid overdose-related EMS runs.…”