Predictive Models May Complement or Provide an Alternative to Existing Strategies for Assessing the Enteric Pathogen Contamination Status of Northeastern Streams Used to Provide Water for Produce Production

Abstract

While the Food Safety Modernization Act established standards for the use of surface water for produce production, water quality is known to vary over space and time. Targeted approaches for identifying hazards in water that account for this variation may improve growers' ability to address pre-harvest food safety risks. Models that utilize publicly-available data (e.g., land-use, real-time weather) may be useful for developing these approaches. The objective of this study was to use pre-existing datasets collected in 2017 (N = 181 samples) and 2018 (N = 191 samples) to train and test models that predict the likelihood of detecting Salmonella and pathogenic E. coli markers (eaeA, stx) in agricultural water. Four types of features were used to train the models: microbial, physicochemical, spatial and weather. "Full models" were built using all four features types, while "nested models" were built using between one and three types. Twenty learners were used to develop separate full models for each pathogen. Separately, to assess information gain associated with using different feature types, six learners were randomly selected and used to develop nine, nested models each. Performance measures for each model were then calculated and compared against baseline models where E. coli concentration was the sole covariate. In the methods, we outline the advantages and disadvantages of each learner. Overall, full models built using ensemble (e.g., Node Harvest) and "blackbox" (e.g., SVMs) learners out-performed full models built using more interpretable learners (e.g., tree-and rule-based learners) for both outcomes. However, nested eaeAstx models built using interpretable learners and microbial data performed almost as well as these full models. While none of the nested Salmonella models performed as well as the full models, nested models built using spatial data consistently outperformed models that excluded spatial data. These findings demonstrate that machine learning approaches can be used to predict when and where pathogens are likely to be present in agricultural water. This study serves as a proof-of-concept that can be built upon once larger datasets become available and provides guidance on Weller et al. Predicting Foodborne Pathogen Presence in Water the learner-data combinations that should be the foci of future efforts (e.g., tree-based microbial models for pathogenic E. coli).

Interpretability Versus Accuracy: A Comparison of Machine Learning Models Built Using Different Algorithms, Performance Measures, and Features to Predict E. coli Levels in Agricultural Water

“…By using a continuous outcome, the present study complements a recent publication that focused on binary, categorical outcomes [detection/non-detection of enteric pathogens (Weller et al, 2020a)]. To the authors' knowledge, this is also the first study to compare the performance of models for predicting E. coli levels in agricultural water that were built using different feature types (i.e., geospatial, physicochemical water quality features, stream traits, and weather).…”

“…Spatial data were obtained from publicly available sources and analyzed using ArcGIS version 10.2 and R version 3.5.3. Briefly, the inverse-distance weighted (IDW) proportion of cropland, developed land, forest-wetland cover, open water, and pasture land for each watershed as well as the floodplain and stream corridor upstream of each sampling site was calculated as previously described [ (King et al, 2005;Weller et al, 2020a; Supplementary Table S1). In addition to characterizing land cover, we also determined if specific features were present in each watershed.…”

“…Hyperparameter tuning was performed using 3fold cross-validation repeated 10 times. Tuning was performed Kuhn and Johnson (2016) and Weller et al, (2020a) to i) reflect the algorithms used here, and ii) report information relevant to continuous (as opposed to categorical) data a,b (Hothorn et al, 2006;Strobl et al, 2007a;Strobl et al, 2008;Strobl et al, 2009)…”

“…Three tree-based algorithms were used: regression trees (CART), conditional inference trees (CTree), and evolutionary optimal trees (evTree) as described in a previous study focused on predicting pathogen presence in agricultural water (Weller et al, 2020a). Briefly, the three tree-based algorithms were implemented using the rpart (Therneau and Atkinson, 2019), party (Hothorn et al, 2006;Strobl et al, 2007b;Strobl et al, 2008;Strobl et al, 2009), and evtree (Grubinger et al, 2014) packages, respectively.…”

“…While past studies have shown that predictive models can be useful for assessing public health hazards in recreational water (Olyphant, 2005;Hou et al, 2006;Hamilton and Luffman, 2009;Francy et al, 2013;Francy et al, 2014;Dada and Hamilton, 2016;Dada, 2019;Rossi et al, 2020), no models, to the author's knowledge, have been developed to predict E. coli levels in surface water used for produce production (e.g., for irrigation, pesticide application, dust abatement, frost protection). Moreover, many of the recreational water quality studies only considered one algorithm during model development (e.g., (Olyphant, 2005;Hamilton and Luffman, 2009), including algorithms [e.g., regression, (Olyphant, 2005;Hamilton and Luffman, 2009)], which has more assumptions and may be less accurate than alternate algorithms (e.g., ensemble methods, support vector machines, (Kuhn and Johnson, 2016;Weller et al, 2020a)). As such, there is limited data on 1) how models for predicting E. coli levels in agricultural water should be implemented and validated, or 2) how the data used to train these models should be collected (e.g., types of features to focus data collection efforts on).…”