Using photographic records to quantify accuracy of bird identifications in citizen science data

Abstract

Citizen science data are increasingly used for biodiversity monitoring. However, concerns are often raised over the accuracy of species identifications in citizen science databases, as data are collected mostly by non-professionals. Misidentifications can simultaneously generate two error types: false positives (erroneous reports of a species) and false negatives (lack of reports of the misidentified species). Large-scale assessments of identification errors should provide insights into the strengths and weaknesses of citizen science data. Here we show that citizen science photographic data for birds are trustworthy overall, although problems arise in hard-to-identify bird groups. We reviewed over 104 000 images of 377 passerine species from the southern Neotropics (Argentina) stored in eBirda large citizen science platformand quantified erroneous reports to calculate precision and recall metrics as measures for data accuracy. Precision increases with fewer false positives and recall increases with fewer false negatives; hence, high values of precision and recall will mirror a higher data accuracy. We found that 97% of the photos of all species were correctly identified. Most species (77%; n = 291) showed high accuracy in their identifications (precision and recall > 95%), with 122 species showing no errors. A few hard-to-identify species (10%; n = 40) showed low levels of data quality (63-90% precision or recall). Similarly, few species (12%; n = 46) exhibited intermediate precision or recall scores (90-95%). Further, we uncovered the existence of a complex network of cross-identifications composed of 272 species, with a predominance of tyrant flycatchers and ovenbirds, reflecting the strong traffic of errors that occurs within these families. To our knowledge, our study provides the first large-scale quantification of identification errors in photos submitted by citizen science contributors. We underscore the relevance of performing such assessments to understand how identification errors are distributed across a database before analysing data, and provide tools for citizen science stakeholders to direct more specific efforts toward species that need an improvement in data quality.

Hidden in plain sight: unveiling the distributions of green‐winged grasshoppers (Aiolopus spp.) with citizen‐science data

“…The use of measurements taken from photographs allowed us to provide a robust approach in species' assignment, as already common for individual‐based recognition methods (De Lutio et al., 2021; Ferreira et al., 2020), a factor that is key to the use of citizen‐science records, particularly in the case of cryptic species (Gorleri et al., 2022). We acknowledge that our approach is only applicable to cryptic species with identifiable morphological differentiation that correspond to levels 2 and 3 of the classification proposed by Chenuil et al., 2019.…”

“…Citizen – or community – science is increasingly emerging as a key approach for biodiversity data collection, fostering the study of wildlife worldwide, and namely addressing the urge of clarifying and understanding species' distributions for ecological studies and conservation (Campanaro et al., 2017). The reliability of citizen‐science derived data is often questioned, yet numerous studies indicate that citizen science can provide high‐quality data, comparable to those collected by trained scientists (Aceves‐Bueno et al., 2017), even though their applicability to the study of cryptic species remains limited (Gorleri et al., 2022).…”

Are Lay People Able to Estimate Breeding Bird Diversity?

“…As a rule, the data are not the result of systematic studies and are made by an array of observants of different skill levels. Callaghan, et al [ 38 ] showed that CS data are usually biased towards larger and more common species, which are mostly correctly identified [ 29 ]. Therefore, for specific scientific questions, CS data may be sufficient.…”

“…Most lists of CS data have also been found to be mostly complete at the family level when compared to professional data and provide prompt records for new emerging species [ 28 ]. Since most species are correctly identified as well [ 29 ], this means CS data can be a good predictor for species richness [ 20 ]. Robinson, et al [ 30 ] showed a count difference biased toward undercounts of contributors to the CS platform eBird compared to a professional ornithologist.…”

Characterizing personalized ecologies

“…Another component of capability that has attracted research attention is the ability of people to identify different kinds of animals (and plants). In general, people's identification abilities have been found to be highly variable but on average quite poor, especially in wealthier countries (e.g., Austen et al., 2016; Bashan et al., 2021; Dallimer et al., 2012; Härtel et al., 2023; Pilgrim et al., 2008; Prokop & Rodak, 2009; Robinson et al., 2016; Shwartz et al., 2023; Vaughn et al., 2022), although very much better for those for whom such skills are more important (e.g., Gorleri et al., 2023; Scott & Hallam, 2002). There are some significant qualifiers.…”