Using ensemble methods to improve the robustness of deep learning for image classification in marine environments

Abstract

Recent advances in deep learning are having a profound impact on human civilisation, permeating every aspect of our daily lives (Makridakis, 2017). So too is the impact deep learning is having on the field of ecology, enabling the rapid analysis of vast amounts of data, the likes we have not seen before (Brandt et al., 2020). At a time where we are seeing rapid decline of health and biodiversity of terrestrial and marine environments ('IPCC Special Report: Global Warming of 1.5°C', 2018), the presence of this technology offers a glimmer of hope for the rapid analysis at the grand scale we need to enable intervention and detect habitat change.

Demystifying image-based machine learning: A practical guide to automated analysis of field imagery using modern machine learning tools

“…As previously suggested (Katija et al 2022), these findings imply that distribution shift occur in the FathomNet dataset, and that these shifts can significantly degrade performance when models are trained on data from one set of locations or time periods and deployed on imagery from new locations or time periods. This phenomenon appears to be widespread in imagery collected in the field (Schneider et al 2020, Wyatt et al 2022.…”

“…Typically, this is done to produce test sets that are more representative of new data on which the ML pipeline is intended to be used. For example, if one wishes to train an image classifier to classify coral species from images (Wyatt et al 2022), and this classifier is intended to be used at new locations in the future, one way to test its performance would be to divide the annotated imagery available into distinct spatial locations, and to construct the training and validation set from a subset of those locations, while holding out other locations that the model never sees during training. This type model evaluation seeks to determine whether models are capable of performing well on images that may have very different statistics than the images on which they were trained.…”

“…Out-of-domain performance: evidence for distribution shifts. As noted above, many researchers who wish to use machine learning pipelines to analyze imagery from the field often intend to use trained pipelines to analyze new imagery taken at later dates or different physical locations, rather than focusing solely on images taken from the same database used to construct the training set (Wyatt et al 2022, Beery et al 2018. To simulate this scenario, we performed a nonrandom, four-fold crossvalidation procedure on the overall set of annotated imagery available on our classes of interest in FathomNet.…”

“…A distribution shift occurs when the imagery on which an ML pipeline is trained differs in some systematic way from the imagery on which the pipeline is deployedthat is, the new imagery the ML method is being used to analyze. The term "distribution shift" refers to a generic set of differences that may occur between one set of images (the "in-domain" set) and another (the "out-of-domain" set), including things like differences in lighting, camera attributes, image scene statistics, background clutter, turbidity, and the relative abundances and appearances of different classes of objects (Taori et al 2020, Scoulding et al 2022, Wyatt et al 2022. This problem arises often in scientific sampling, because in this setting, the goal is often to extract data from one image set collected at a particular place and time, and to compare it to data from another image data set collected at some other place or time.…”

“…A wide array of methods have been proposed to improve the performance of ML models on new imagery that is distribution shifted relative to training images. These range from training interventions like digitally altering (i.e., "augmenting") training imagery to destroy irrelevant features that can result in overtraining (Busalaev et al 2020, Zoph et al 2020, to the use of more robust inference frameworks such as ensemble models, which combine predictions of multiple machine learning models (Wyatt et al 2022). To provide a sense for how some of these methods work, and how one might apply them to improve model performance, we applied a suite of training interventions to our case study dataset.…”