Let's Agree to Disagree: Comparing Auto-Acoustic Identification Programs for Northeastern Bats

Abstract

With the declines in abundance and changing distribution of white-nose syndrome–affected bat species, increased reliance on acoustic monitoring is now the new “normal.” As such, the ability to accurately identify individual bat species with acoustic identification programs has become increasingly important. We assessed rates of disagreement between the three U.S. Fish and Wildlife Service–approved acoustic identification software programs (Kaleidoscope Pro 4.2.0, Echoclass 3.1, and Bat Call Identification 2.7d) and manual visual identification using acoustic data collected during summers from 2003 to 2017 at Fort Drum, New York. We assessed the percentage of agreement between programs through pairwise comparisons on a total nightly count level, individual file level (e.g., individual echolocation pass call file), and grouped maximum likelihood estimate level (e.g., probability values that a species is misclassified as present when in fact it is absent) using preplanned contrasts, Akaike Information Criterion, and annual confusion matrices. Interprogram agreement on an individual file level was low, as measured by Cohen's Kappa (0.2–0.6). However, site-night level pairwise comparative analysis indicated that program agreement was higher (40–90%) using single season occupancy metrics. In comparing analytical outcomes of our different datasets (i.e., how comparable programs and visual identification are regarding the relationship between environmental conditions and bat activity), we determined high levels of congruency in the relative rankings of the model as well as the relative level of support for each individual model. This indicated that among individual software packages, when analyzing bat calls, there was consistent ecological inference beyond the file-by-file level at the scales used by managers. Depending on objectives, we believe our results can help users choose automated software and maximum likelihood estimate thresholds more appropriate for their needs and allow for better cross-comparison of studies using different automated acoustic software.

Occupancy and Detectability of Northern Long‐eared Bats in the Lake States Region

“…Though nightly agreement rates were low (35%), both software programs identified northern long‐eared bats on at least one night given the species was present for most sites. Nocera et al (2019) similarly found low to medium nightly file‐by‐file agreement rates between versions of the software programs we used, but concluded that findings did not differ over the course of an entire night in an occupancy framework. The probability a true positive was classified as certain was low.…”

“…Low probability of false positive detections indicates that leaving detectors on the landscape for ≥10 days and using results from both automated bat call identification programs results in acceptable rates of false positives. Moreover, our false‐positive results provide a lens through which managers may be able to evaluate analytical results derived from these 2 programs at their own sites as well as other published studies within this region (Nocera et al 2019). These low false‐positive detection rates provide us with greater confidence that the species was truly present at a site and therefore, stronger inference about the occupancy and detection relationships relative to habitat and temporal factors.…”

White‐nose Syndrome and Environmental Correlates to Landscape‐Scale Bat Presence

“…We modeled nightly bat presence for each species using binary-response generalized linear mixed models (GLMMs) in program R version 3.6.0 (R Core Team 2019) with package glmmTMB (Brooks et al 2017) with nested random effects of the 3 sampling sites within each of the larger 10 study areas to account for spatial autocorrelation. We chose a binomial approach because post WNS, overall nightly bat activity is low in many areas (Nocera et al 2019a) and data automated software identification with a set MLE trigger has shown to be conservative yet robust (Nocera et al 2019b). We used a 2-step information theoretic approach in building the candidate models with Akaike's Information Criterion corrected for small sample size (AICc from package bbmle, Bolker 2017; Burnham and Anderson 2002) to rank models and then considered best supported models with a ΔAICc < 2.…”

Effects of Environmental Clutter on Synthesized Chiropteran Echolocation Signals in an Anechoic Chamber

“…Acoustic recording of bats is a widely used monitoring tool for assessing occupancy and relative activity of bat communities. Acoustic surveys have become increasingly important with the advent of white-nose syndrome (WNS) and the expansion of wind energy in North America that are significant mortality factors, because the capture of rare bats has become time-intensive and cost-prohibitive [1]. Ultrasonic acoustic lures also are an emerging technology that shows promise to attract bats during mist-net surveys in part to overcome decreases in catch-per-unit effort from WNS [2][3][4][5].…”

“…Bat detectors generally are placed in open, uncluttered areas such as fields, forest canopy gap openings, stream corridors, and unimproved roads to record search phase echolocation pulses that exhibit the most interspecific variation and the least amount of intraspecific variation [11,[16][17][18]. This maximizes echolocation classification accuracy with automatic identification programs because many program classifiers are built using voucher calls recorded from hand releases, flight cage recordings, and zip line restraint of bats in open, uncluttered areas [1,19]. Typically, commercially available automatic identification programs have not intentionally incorporated clutter-recorded calls into their classifiers [1,16].…”