The impacts of fine-tuning, phylogenetic distance, and sample size on big-data bioacoustics

Abstract

Vocalizations in animals, particularly birds, are critically important behaviors that influence their reproductive fitness. While recordings of bioacoustic data have been captured and stored in collections for decades, the automated extraction of data from these recordings has only recently been facilitated by artificial intelligence methods. These have yet to be evaluated with respect to accuracy of different automation strategies and features. Here, we use a recently published machine learning framework to extract syllables from ten bird species ranging in their phylogenetic relatedness from 1 to 85 million years, to compare how phylogenetic relatedness influences accuracy. We also evaluate the utility of applying trained models to novel species. Our results indicate that model performance is best on conspecifics, with accuracy progressively decreasing as phylogenetic distance increases between taxa. However, we also find that the application of models trained on multiple distantly related species can improve the overall accuracy to levels near that of training and analyzing a model on the same species. When planning big-data bioacoustics studies, care must be taken in sample design to maximize sample size and minimize human labor without sacrificing accuracy.

Machine learning reveals relationships between song, climate, and migration in coastalZonotrichia leucophrys

“…All selections were output as TXT files. We then split the original recordings and their corresponding annotation files into subsets that had approximately equal amounts of annotated syllables and silence using previously generated scripts (Provost et al 2022). Although TweetyNet can identify and segment multiple kinds of syllables, we annotated all syllables as “1”.…”

“…We then fine-tuned this model by continuing to train it on just data from nuttalli and pugetensis . We used vak version 0.6.0 and TweetyNet version 0.8.0 for that and we used identical parameters to (Provost et al 2022). After being fine-tuned, we had the model predict the locations of syllables for all data, including data that TweetyNet had never seen before.…”

“…Machine learning has also been applied to syllable segmentation with high accuracy via the neural network TweetyNet (Cohen et al 2020). Originally trained on single species, the algorithm has since been applied to a suite of birds (Provost et al 2022) with similarly high performance, allowing for the use of larger datasets. Although the segmentation itself has been sped up, it is unknown whether this and other automatic segmentation methods effectively recapitulate known differences in populations.…”

“…To automatically annotate the White-crowned Sparrow songs, we used the TweetyNet model trained on Zonotrichia leucophrys from Provost et al (2022). The latter analyses included examples from all five subspecies.…”

Crowsetta: A Python tool to work with any format for annotating animal vocalizations and bioacoustics data.

“…Crowsetta made it possible to work with several annotation formats when using vak to benchmark a neural network architecture that automates annotation of birdsong, TweetyNet Cohen & Nicholson, 2023). Since then, crowsetta has been used in tandem with vak by several research groups in neuroscience (Goffinet et al, 2021;McGregor et al, 2022) and bioacoustics (Provost et al, 2022).…”