A practical tool for public health surveillance: Semi-automated coding of short injury narratives from large administrative databases using Naïve Bayes algorithms

Abstract

Public health surveillance programs in the U.S. are undergoing landmark changes with the availability of electronic health records and advancements in information technology. Injury narratives gathered from hospital records, workers compensation claims or national surveys can be very useful for identifying antecedents to injury or emerging risks. However, classifying narratives manually can become prohibitive for large datasets. The purpose of this study was to develop a human-machine system that could be relatively easily tailored to routinely and accurately classify injury narratives from large administrative databases such as workers compensation. We used a semi-automated approach based on two Naïve Bayesian algorithms to classify 15,000 workers compensation narratives into two-digit Bureau of Labor Statistics (BLS) event (leading to injury) codes. Narratives were filtered out for manual review if the algorithms disagreed or made weak predictions. This approach resulted in an overall accuracy of 87%, with consistently high positive predictive values across all two-digit BLS event categories including the very small categories (e.g., exposure to noise, needle sticks). The Naïve Bayes algorithms were able to identify and accurately machine code most narratives leaving only 32% (4853) for manual review. This strategy substantially reduces the need for resources compared with manual review alone.

Harnessing information from injury narratives in the ‘big data’ era: understanding and applying machine learning for injury surveillance

“…The authors 26 found while the overall sensitivity of the two independent models was fairly good (0.67 naive sw , 0.65 naive seq ), both algorithms independently predicted some categories much better than others, skewing the final distribution of the coded data (χ² p<0.0001), and most of the cases in the smaller categories were not found. The sequence-word model showed improved performance where word order was important for differentiating causality.…”

“…A sample of WC claims incident narratives with BLS OIICS code assignments are shown below: The occurrence or probability of each word in each category (Pn j /C i ) was calculated as well as the marginal probability of each event category in the training data set (P(C i )); these are the two parameters necessary for the reduced Naïve Bayes algorithm. 26 These statistics calculated from the training narratives were stored in a probability table and used to train the Free text search of motor vehicle insurance claims database for 4 years to identify claims where road work occurring and key word categorisation of precrash activities and crash types through word frequency count and manual grouping of similar words to prepare keyword search strategy. Expanded to test a Bayesian modelling approach in second paper First paper identified number of incidents and categorised precrash activities and crash types to examine patterns of incidents.…”

“…However, this may sacrifice accuracy, with the authors' preliminary work using Naïve Bayes, LR and SVM showing that these preprocessing approaches have the potential to reduce the overall detection (distinguishing between categories) capability, and especially for small categories. 43 Part of the problem is that such approaches do not Table 2 The accuracy of the human-machine classification system: implementation of a strategic filter † based on agreement between two Naïve Bayes algorithms 26 †A filter is a technique to decide which narratives the computer should classify versus which should be left for a human to read and classify. ‡Gold standard codes were assigned to each narrative by expert manual coders.…”

“…BLS, Bureau of Labor Statistics; OIICS, Occupational Injury and Illness Classification System. *-Asterisks denote a summary level code not assigned to individual cases (see http://www.bls.gov/iif/osh_oiics_2010_2_4_2.pdf) Table 3 The accuracy of the human-machine classification system: implementation of a strategic filter † based on agreement between the two naïve bayes algorithms (results for small categories only, n<100 cases in each category) 26 †A filter is a technique to decide which narratives the computer should classify versus which should be left for a human to read and classify. ‡Gold standard codes were assigned to each narrative by expert manual coders.…”

Comparison of methods for auto-coding causation of injury narratives

“…In fact, within the field of coding injury narratives, Lehto et al (2009) and Marucci-Wellman et al (2011) have considered two-word (and longer) sequences in a separate model referred to as “Fuzzy Bayes.” Also, Grattan et al (2014) and Marucci-Wellman et al (2015) used two-word sequences within the Naïve Bayes framework, however, single-word and two-word sequences were used in separate models, not in a single model. Measure (2014) provides a more exhaustive investigation into which features optimize various auto-coder models and found that both the Naïve Bayes and logistic event auto-coders benefit from including single word and two-word features along with the North American Industry Classification System (NAICS) code of the employing establishment in a single model.…”