Stronger together: Combining automated classifiers with manual post-validation optimizes the workload vs reliability trade-off of species identification in bat acoustic surveys

Abstract

Owing to major technological advances, bioacoustics has become a burgeoning field in ecological research worldwide. Autonomous passive acoustic recorders are becoming widely used to monitor aerial insectivorous bats, and automatic classifiers have emerged to aid researchers in the daunting task of analyzing the resulting massive acoustic datasets.However, the scarcity of comprehensive reference call libraries still hampers their wider application in highly diverse tropical assemblages. Capitalizing on a unique acoustic dataset of more than 650,000 bat call sequences collected over a 3-year period in the Brazilian Amazon, the aims of this study were (a) to assess how pre-identified recordings of free-flying and hand-released bats could be used to train an automatic classification algorithm (random forest), and (b) to optimize acoustic analysis protocols by combining automatic classification with visual post-validation, whereby we evaluated the proportion of sound files to be postvalidated for different thresholds of classification accuracy. Classifiers were trained at species or sonotype (group of species with similar calls) level. Random forest models confirmed the reliability of using calls of both free-flying and hand-released bats to train custom-built automatic classifiers. To achieve a general classification accuracy of ~85%, random forest had to be trained with at least 500 pulses per species/sonotype. For seven out of 20 sonotypes, the most abundant in our dataset, we obtained high classification accuracy (>90%). Adopting a desired accuracy probability threshold of 95% for the random forest classifier, we found that the percentage of sound files required for manual post-validation could be reduced by up to 75%, a significant saving in terms of workload. Combining automatic classification with manual ID through fully customizable classifiers implemented in open-source software as demonstrated here shows great potential to help overcome the acknowledged risks and biases associated with the sole reliance on automatic classification.

Echolocation and Stratum Preference: Key Trait Correlates of Vulnerability of Insectivorous Bats to Tropical Forest Fragmentation

“…Frequency of maximum energy (FME), also known as peak frequency, is the most intense frequency in the call (López-Baucells et al, 2016). Amongst other environmental and acoustic factors, it determines how quickly the intensity of a call will diminish and therefore, will influence the kind of habitat in which the bat will be able to fly and forage.…”

“…Although acoustic methods are the most suitable to sample AIB, the costs and time commitment involved in acoustic surveys is still considerable. Moreover, the echolocation calls of many tropical species and the variation among them have not yet been adequately documented, and reference call libraries for tropical regions are scarce (MacSwiney et al, 2008;López-Baucells et al, 2016). As a result of these limitations, AIB continue to be underrepresented in inventories and ecological studies (Cunto and Bernard, 2012), and there is a lack of data about their vulnerability to habitat fragmentation (but see Estrada-Villegas et al, 2010;Bader et al, 2015a,b).…”

Accounting for automated identification errors in acoustic surveys

“…However, automated identification software has been criticized due to significant error rates, suggesting cautious and limited use (Russo & Voigt, 2016;Rydell, Nyman, Eklöf, Jones, & Russo, 2017), which heavily reduces the advantages of automated algorithms. Nonetheless, authors have highlighted the potential for combining automated classifiers with manual validation to help overcome error risks associated with automated identification, and so saving a huge amount of work in reducing the extent of manual checking required (López-Baucells et al, 2019). Moreover, most available software provides confidence scores associated with each automated identification in the form of probabilities or other numerical indexes (Obrist, Boesch, & Fluckiger, 2004;Waters & Barlow, 2013), which unlike the error rate is not dependent of the relative abundance of the species.…”

Pipistrellus pipistrellus and Pipistrellus pygmaeus in the Iberian Peninsula: An Annotated Segmented Dataset and a Proof of Concept of a Classifier in a Real Environment

“…Nevertheless, the use of automated identification may also be hazardous, as some of those techniques can provide an inaccurate identification of species without testing the libraries in the field and with negative consequences in bat management and conservation [19]. The combination of automatic and manual identification optimizes bat call classification [20]. Therefore, in this paper we propose working with datasets of these two species that have already been manually analyzed to ensure the quality of the developed algorithm.…”

“…There are currently some tools available to automatically detect bats which claim to achieve high reliability results, although they have to be used with caution due to possible false positive evaluations (https://www.wildlifeacoustics.com/products/kaleidoscope-pro, http://ibatsid.euwest-1.elasticbeanstalk.com/file.jsp, http://www.leclub-biotope.com/en/sonochiro/422-sonochiroenglish-version.html [20]). It is highly advisable to use preidentified reference call libraries and to postvalidate and train the accuracy of the algorithms to minimize the risk of automatic classification [20]. Taking this into consideration, we intend to develop a tool to aid researchers with the manual identification of bats.…”