Machine Learning Undercounts Reproductive Organs on Herbarium Specimens but Accurately Derives Their Quantitative Phenological Status: A Case Study of Streptanthus tortuosus

Abstract

Machine learning (ML) can accelerate the extraction of phenological data from herbarium specimens; however, no studies have assessed whether ML-derived phenological data can be used reliably to evaluate ecological patterns. In this study, 709 herbarium specimens representing a widespread annual herb, Streptanthus tortuosus, were scored both manually by human observers and by a mask R-CNN object detection model to (1) evaluate the concordance between ML and manually-derived phenological data and (2) determine whether ML-derived data can be used to reliably assess phenological patterns. The ML model generally underestimated the number of reproductive structures present on each specimen; however, when these counts were used to provide a quantitative estimate of the phenological stage of plants on a given sheet (i.e., the phenological index or PI), the ML and manually-derived PI’s were highly concordant. Moreover, herbarium specimen age had no effect on the estimated PI of a given sheet. Finally, including ML-derived PIs as predictor variables in phenological models produced estimates of the phenological sensitivity of this species to climate, temporal shifts in flowering time, and the rate of phenological progression that are indistinguishable from those produced by models based on data provided by human observers. This study demonstrates that phenological data extracted using machine learning can be used reliably to estimate the phenological stage of herbarium specimens and to detect phenological patterns.

Leveraging machine learning and citizen science data to describe flowering phenology across South Africa

“…Previous work has demonstrated the potential of machine learning algorithms, such as CNNs, in extracting phenological data from digital images, but have generally applied methods to a single or a few closely related species using carefully curated datasets (e.g. Pearson et al ., 2020; Soltis et al ., 2020; Love et al ., 2021; Davis et al ., 2022). Here, we apply a CNN classifier (ResNet-18) to the flora of South Africa, a region famed for its exceptional floristic diversity.…”

Challenges and opportunities in applying AI to evolutionary morphology

“…AI methods, including CNNs, have been successfully applied on small training datasets to recognise species and extract both discrete and meristic traits (Wäldchen and Mäder, 2018). Other examples include using ML tools to extract, classify and count reproductive structures (Goëau et al, 2022;Love et al, 2021), as well as to produce basic measurements such as leaf size (Hussein et al, 2021;Weaver et al, 2020). These methods have also been shown to work on x-ray scans of fossil leaves (Wilf et al, 2021), including counting stomatal and epidermal cells for palaeoclimatic analysis (Zhang et al, 2023).…”

From leaves to labels: Building modular machine learning networks for rapid herbarium specimen analysis with LeafMachine2

“…These frameworks are extremely flexible, well‐supported, and surprisingly approachable. As a result, many recent projects have also coalesced around these two frameworks with great success, including efforts to segment leaves (Younis et al, 2020; Triki et al, 2020, 2021; Guo et al, 2021; Hussein et al, 2021b; Gu et al, 2022; Ott and Lautenschlager, 2022), segment plant tissue (Love et al, 2021; Goëau et al, 2022; Milleville et al, 2023), isolate plant organs (Davis et al, 2020; Pearson et al, 2020; Triki et al, 2020; Ott and Lautenschlager, 2022), extract specimen label data (Milleville et al, 2023), isolate diseased or damaged leaf tissue (Kaur et al, 2022; Mu et al, 2022; Kavitha Lakshmi and Savarimuthu, 2023), measure bird skeletons (Weeks et al, 2023), isolate preserved snakes (Curlis et al, 2022), segment fossils (Panigrahi et al, 2022), or remotely monitor phenology (Mann et al, 2022). However, rather than relying on a single machine learning architecture to extract trait and archival data from specimens, we developed a modular framework of seven different machine learning algorithms that work in tandem to comprehensively process each image (Table 2, Figure 1).…”

“…To overcome these limitations, many groups turned to machine learning algorithms, typically some kind of convolutional neural network (CNN), which can categorize individual pixels as members of discrete classes (Ott et al, 2020; Weaver et al, 2020; Younis et al, 2020; Triki et al, 2020, 2021; Goëau et al, 2020, 2022; Guo et al, 2021; Love et al, 2021; Hussein et al, 2021b; Gu et al, 2022; Ott and Lautenschlager, 2022; Milleville et al, 2023). For the task of isolating and measuring individual leaves, semantic segmentation algorithms still lack the power to resolve complex situations (e.g., overlapping leaves) because they produce one mask that contains all leaf pixels and require postprocessing to obtain usable results (Weaver et al, 2020; Hussein et al, 2021b, 2022).…”