Thinking like a naturalist: enhancing computer vision of citizen science images by harnessing contextual data

Abstract

The accurate identification of species in images submitted by citizen scientists is currently a bottleneck for many data uses. Machine learning tools offer the potential to provide rapid, objective and scalable species identification for the benefit of many aspects of ecological science. Currently, most approaches only make use of image pixel data for classification. However, an experienced naturalist would also use a wide variety of contextual information such as the location and date of recording. Here, we examine the automated identification of ladybird (Coccinellidae) records from the British Isles submitted to the UK Ladybird Survey, a volunteer‐led mass participation recording scheme. Each image is associated with metadata; a date, location and recorder ID, which can be cross‐referenced with other data sources to determine local weather at the time of recording, habitat types and the experience of the observer. We built multi‐input neural network models that synthesize metadata and images to identify records to species level. We show that machine learning models can effectively harness contextual information to improve the interpretation of images. Against an image‐only baseline of 48.2%, we observe a 9.1 percentage‐point improvement in top‐1 accuracy with a multi‐input model compared to only a 3.6% increase when using an ensemble of image and metadata models. This suggests that contextual data are being used to interpret an image, beyond just providing a prior expectation. We show that our neural network models appear to be utilizing similar pieces of evidence as human naturalists to make identifications. Metadata is a key tool for human naturalists. We show it can also be harnessed by computer vision systems. Contextualization offers considerable extra information, particularly for challenging species, even within small and relatively homogeneous areas such as the British Isles. Although complex relationships between disparate sources of information can be profitably interpreted by simple neural network architectures, there is likely considerable room for further progress. Contextualizing images has the potential to lead to a step change in the accuracy of automated identification tools, with considerable benefits for large‐scale verification of submitted records.

Revisiting animal photo‐identification using deep metric learning and network analysis

“…Since it is suitable to any species, further data analysis on other species will help answer this question. However, additional strategies would help including the integration of contextual information (Beery et al., 2019; Terry et al., 2020) such as time, GPS positioning or animal social context. Using accurate segmentation of animal body (Brodrick et al., 2019; He et al., 2017) will undoubtedly be a solution against side effects of rectangular cropping.…”

Robust and simplified machine learning identification of pitfall trap‐collected ground beetles at the continental scale

“…Incorporating known species ranges improves model predictions, but prevents the ability to detect species that have expanded their ranges. One solution may be to include contextual metadata as part of the image feature vectors (Terry et al., 2020). Contextual metadata could include primary metadata such as location and date, as well as secondary metadata (if collected from external data sources) such as weather and habitat type (Terry et al., 2020).…”

“…One solution may be to include contextual metadata as part of the image feature vectors (Terry et al., 2020). Contextual metadata could include primary metadata such as location and date, as well as secondary metadata (if collected from external data sources) such as weather and habitat type (Terry et al., 2020). The models can then weigh contextual metadata against image features when making a prediction.…”

Revisiting animal photo-identification using deep metric learning and network analysis

“…Since it is suitable to any species, further data analysis on other species will help answer this question. However, additional strategies would help including the integration of contextual information (Beery et al ., 2019; Terry et al ., 2020) such as time, GPS positioning or animal social context. Using accurate segmentation of animal body (He et al ., 2017; Brodrick et al ., 2019) will undoubtedly be a solution against side effects of rectangular cropping.…”