Interpretability Versus Accuracy: A Comparison of Machine Learning Models Built Using Different Algorithms, Performance Measures, and Features to Predict E. coli Levels in Agricultural Water

Abstract

Since E. coli is considered a fecal indicator in surface water, government water quality standards and industry guidance often rely on E. coli monitoring to identify when there is an increased risk of pathogen contamination of water used for produce production (e.g., for irrigation). However, studies have indicated that E. coli testing can present an economic burden to growers and that time lags between sampling and obtaining results may reduce the utility of these data. Models that predict E. coli levels in agricultural water may provide a mechanism for overcoming these obstacles. Thus, this proof-of-concept study uses previously published datasets to train, test, and compare E. coli predictive models using multiple algorithms and performance measures. Since the collection of different feature data carries specific costs for growers, predictive performance was compared for models built using different feature types [geospatial, water quality, stream traits, and/or weather features]. Model performance was assessed against baseline regression models. Model performance varied considerably with root-mean-squared errors and Kendall’s Tau ranging between 0.37 and 1.03, and 0.07 and 0.55, respectively. Overall, models that included turbidity, rain, and temperature outperformed all other models regardless of the algorithm used. Turbidity and weather factors were also found to drive model accuracy even when other feature types were included in the model. These findings confirm previous conclusions that machine learning models may be useful for predicting when, where, and at what level E. coli (and associated hazards) are likely to be present in preharvest agricultural water sources. This study also identifies specific algorithm-predictor combinations that should be the foci of future efforts to develop deployable models (i.e., models that can be used to guide on-farm decision-making and risk mitigation). When deploying E. coli predictive models in the field, it is important to note that past research indicates an inconsistent relationship between E. coli levels and foodborne pathogen presence. Thus, models that predict E. coli levels in agricultural water may be useful for assessing fecal contamination status and ensuring compliance with regulations but should not be used to assess the risk that specific pathogens of concern (e.g., Salmonella, Listeria) are present.

Prediction of E. coli Concentrations in Agricultural Pond Waters: Application and Comparison of Machine Learning Algorithms

“…While model performance generally did not significantly differ, the RF model was found to provide consistently better performance than any of the other models evaluated. Several publications focusing on ML model evaluation specifically for microbial water quality purposes have reached similar conclusions (Avila et al, 2018;Chen et al, 2020;Weller et al, 2021). The SGB model was found to provide the second-best performance across all three metrics from all datasets.…”

“…The SVM algorithm typically provided lower RMSE than the kNN algorithm and this may be because SVM is robust to outliers especially when non-linear kernels are used. The RBF kernel was used in this study because it provided substantially better results than the linear or polynomial kernels (data not shown) which was also reported in the work by Weller et al (2021) who used ML algorithms for the prediction of E. coli in NY streams. Several other water quality studies have also reported better performance of SVM than kNN when the two have been compared (Modaresi and Araghinejad, 2014;Danades et al, 2016;Babbar and Babbar, 2017;Prakash et al, 2018;Chen et al, 2020).…”

“…This example highlights the importance on the choice of performance metric reported in algorithm evaluation studies and the advantage of using multiple performance metrics. We compared only five different algorithms in this study which were chosen due their popularity, but many more ML algorithms and their modifications exist and can be tested for regression-type application (Kuhn, 2008;Kuhn and Johnson, 2013;Weller et al, 2021). We chose not to run artificial neural networks (ANN) due to constraints of the dataset dimensions but other researchers have found success in applying ANN algorithms in the field of microbial water quality (Motamarri and Boccelli, 2012;Buyrukoglu et al, 2021).…”

Comparison of Resampling Algorithms to Address Class Imbalance when Developing Machine Learning Models to Predict Foodborne Pathogen Presence in Agricultural Water

“…Conversely, the current study focuses on the 1) development and comparison of predictive models using different algorithms and feature types, 2) impact of resampling methods (to address data imbalance) on model performance, and 3) identification of features that drive model accuracy. Moreover, unlike previous, applied studies that developed models to predict the presence of food safety hazards in agricultural water using balanced presenceabsence Polat et al (2019), Weller et al (2020c) or continuous Weller et al (2021) data, the current study focuses on predicting Listeria contamination using moderately (nonpathogenic Listeria spp.) and severely (L. monocytogenes) imbalanced presenceabsence data (Table 1).…”

“…Models built using all four feature types were designated "full models." The 15 learners used to build the full models can be grouped into 1) tree-based learners, 2) ensemble learners (or forests), 3) regression, rule-based learners, and 4) support vector machines [SVM; for descriptions of each learner as well as its (dis)advantages and tunable parameters see (Bischl et al, 2016b;Kuhn and Johnson, 2016;Weller et al, 2020c;Weller et al, 2021)]. Separately from the full models, "nested models" were developed to assess the relative information gain associated with using different feature types for model training.…”