Comparison of Resampling Algorithms to Address Class Imbalance when Developing Machine Learning Models to Predict Foodborne Pathogen Presence in Agricultural Water

Abstract

Recent studies have shown that predictive models can supplement or provide alternatives to E. coli-testing for assessing the potential presence of food safety hazards in water used for produce production. However, these studies used balanced training data and focused on enteric pathogens. As such, research is needed to determine 1) if predictive models can be used to assess Listeria contamination of agricultural water, and 2) how resampling (to deal with imbalanced data) affects performance of these models. To address these knowledge gaps, this study developed models that predict nonpathogenic Listeria spp. (excluding L. monocytogenes) and L. monocytogenes presence in agricultural water using various combinations of learner (e.g., random forest, regression), feature type, and resampling method (none, oversampling, SMOTE). Four feature types were used in model training: microbial, physicochemical, spatial, and weather. “Full models” were trained using all four feature types, while “nested models” used between one and three types. In total, 45 full (15 learners*3 resampling approaches) and 108 nested (5 learners*9 feature sets*3 resampling approaches) models were trained per outcome. Model performance was compared against baseline models where E. coli concentration was the sole predictor. Overall, the machine learning models outperformed the baseline E. coli models, with random forests outperforming models built using other learners (e.g., rule-based learners). Resampling produced more accurate models than not resampling, with SMOTE models outperforming, on average, oversampling models. Regardless of resampling method, spatial and physicochemical water quality features drove accurate predictions for the nonpathogenic Listeria spp. and L. monocytogenes models, respectively. Overall, these findings 1) illustrate the need for alternatives to existing E. coli-based monitoring programs for assessing agricultural water for the presence of potential food safety hazards, and 2) suggest that predictive models may be one such alternative. Moreover, these findings provide a conceptual framework for how such models can be developed in the future with the ultimate aim of developing models that can be integrated into on-farm risk management programs. For example, future studies should consider using random forest learners, SMOTE resampling, and spatial features to develop models to predict the presence of foodborne pathogens, such as L. monocytogenes, in agricultural water when the training data is imbalanced.

Salmonella Prevalence Is Strongly Associated with Spatial Factors while Listeria monocytogenes Prevalence Is Strongly Associated with Temporal Factors on Virginia Produce Farms

“…Since growing environments are complex, and past studies have shown that environmental factors can interact to affect odds of pathogen detection, conditional forest analysis was used to characterize hierarchical associations between odds of Salmonella, L, LM, and LS isolation and environmental factors using the mlr and party packages, as previously described (71,72). Briefly, a 3-fold cross-validation repeated 10 times was used to tune the number of variables considered in each split, to determine the maximum depth of the tree, and to assess model fit.…”

“…Briefly, a 3-fold cross-validation repeated 10 times was used to tune the number of variables considered in each split, to determine the maximum depth of the tree, and to assess model fit. The synthetic minority oversampling technique (SMOTE) for resampling was used to address class imbalance in the outcomes (72). The relative strength of associations between each covariate and odds of isolating the microbial target were quantified using conditional variable importance.…”

Factors Influencing the Chance of Inheriting the Family Farming Career among Heirs in the Upper Northern Region of Thailand in the Crisis of Farming Labor Decline

“…Since the error of prediction in the minority class is dominated by the error of prediction in the majority class due to the data size difference, the accuracy of prediction in the minority class is obviously poorer than the accuracy of the majority class. The resampling method has been proven to be a feasible solution to this problem [49]. In this paper, we propose a method that can be used to select additional data from the minority class.…”

Humans and Hoofed Livestock Are the Main Sources of Fecal Contamination of Rivers Used for Crop Irrigation: A Microbial Source Tracking Approach

“…Briefly, hyperparameters were tuned, models trained, and performance assessed using three-fold cross-validation repeated 10 times; cross-validation was performed to optimize R-squared (for continuous outcomes) and area under the curve (AUC; for binary outcomes). For imbalanced, binary outcomes (i.e., Giardia detection versus non-detection), SMOTE resampling was performed as part of tuning (Kuhn and Johnson, 2016;Weller et al, 2021).…”