Using Convolutional Neural Networks to Efficiently Extract Immense Phenological Data From Community Science Images

Abstract

Community science image libraries offer a massive, but largely untapped, source of observational data for phenological research. The iNaturalist platform offers a particularly rich archive, containing more than 49 million verifiable, georeferenced, open access images, encompassing seven continents and over 278,000 species. A critical limitation preventing scientists from taking full advantage of this rich data source is labor. Each image must be manually inspected and categorized by phenophase, which is both time-intensive and costly. Consequently, researchers may only be able to use a subset of the total number of images available in the database. While iNaturalist has the potential to yield enough data for high-resolution and spatially extensive studies, it requires more efficient tools for phenological data extraction. A promising solution is automation of the image annotation process using deep learning. Recent innovations in deep learning have made these open-source tools accessible to a general research audience. However, it is unknown whether deep learning tools can accurately and efficiently annotate phenophases in community science images. Here, we train a convolutional neural network (CNN) to annotate images of Alliaria petiolata into distinct phenophases from iNaturalist and compare the performance of the model with non-expert human annotators. We demonstrate that researchers can successfully employ deep learning techniques to extract phenological information from community science images. A CNN classified two-stage phenology (flowering and non-flowering) with 95.9% accuracy and classified four-stage phenology (vegetative, budding, flowering, and fruiting) with 86.4% accuracy. The overall accuracy of the CNN did not differ from humans (p = 0.383), although performance varied across phenophases. We found that a primary challenge of using deep learning for image annotation was not related to the model itself, but instead in the quality of the community science images. Up to 4% of A. petiolata images in iNaturalist were taken from an improper distance, were physically manipulated, or were digitally altered, which limited both human and machine annotators in accurately classifying phenology. Thus, we provide a list of photography guidelines that could be included in community science platforms to inform community scientists in the best practices for creating images that facilitate phenological analysis.

Photographs as an essential biodiversity resource: drivers of gaps in the vascular plant photographic record

“…New Phytologist (2023) 238: 1685-1694 www.newphytologist.com Ó 2023 The Authors New Phytologist Ó 2023 New Phytologist Foundation likely to then facilitate future identification by both professional and citizen scientists, as well as the improvement of automated species identification and phenological data extraction through the use of, for example, deep convolutional neural networks (M€ ader et al, 2021;Reeb et al, 2022;Yang et al, 2022), leading to more verified photographs and better data on spatial-temporal distributions and phenology. The development of a centralised photographic repository may improve the discoverability of online plant photographs (Pitman et al, 2021) and better facilitate efforts to complete the photographic record for underrepresented species.…”

“…Efforts to improve the vascular plant photographic record should therefore focus not only on unphotographed species, but also poorly represented species, and should aim to both increase the number of photographs for each species and capture a standardised set of diagnostic features to maximise identifiability (Baskauf & Kirchoff, 2008; Gómez‐Bellver et al ., 2019; Rzanny et al ., 2019, 2022). This verified set of photographs is likely to then facilitate future identification by both professional and citizen scientists, as well as the improvement of automated species identification and phenological data extraction through the use of, for example, deep convolutional neural networks (Mäder et al ., 2021; Reeb et al ., 2022; Yang et al ., 2022), leading to more verified photographs and better data on spatial–temporal distributions and phenology. The development of a centralised photographic repository may improve the discoverability of online plant photographs (Pitman et al ., 2021) and better facilitate efforts to complete the photographic record for underrepresented species.…”

Deep Learning in Plant Phenological Research: A Systematic Literature Review

“…A first, recently published study shows deep learning technologies can be successfully used to extract phenological information from citizen science images. A CNN classified a two-stage phenology (flowering and non-flowering) with 95.9% accuracy and a four-stage phenology (vegetative, budding, flowering, and fruiting) with 86.4% accuracy based on Alliaria petiolata (Reeb et al, 2022 ). As also the studies on the herbarium specimens showed, such annotations can also be done automatically with deep learning algorithms.…”

The use of photos to investigate ecological change

“…Mesaglio & Callaghan, 2021), the identification of functional traits (e.g. Li et al, 2020), or the annotation of plant phenophases (Reeb et al, 2022). In contrast to these object-oriented processing techniques, ecological change can also be assessed using pixel-based change detection algorithms.…”