DeepAction: a MATLAB toolbox for automated classification of animal behavior in video

Abstract

The identification of animal behavior in video is a critical but time-consuming task in many areas of research. Here, we introduce DeepAction, a deep learning-based toolbox for automatically annotating animal behavior in video. Our approach uses features extracted from raw video frames by a pretrained convolutional neural network to train a recurrent neural network classifier. We evaluate the classifier on two benchmark rodent datasets and one octopus dataset. We show that it achieves high accuracy, requires little training data, and surpasses both human agreement and most comparable existing methods. We also create a confidence score for classifier output, and show that our method provides an accurate estimate of classifier performance and reduces the time required by human annotators to review and correct automatically-produced annotations. We release our system and accompanying annotation interface as an open-source MATLAB toolbox.

SaLSa: a combinatory approach of semi-automatic labeling and long short-term memory to classify behavioral syllables

“…Over the last decade, a range of approaches has been developed to classify behavioral syllables (1,3,4,5,8,13,14,15,16,18). SaLSa is the first approach to utilize LSTM for this purpose, to the best of our knowledge.…”

“…Deep learning-based approaches have increasingly been adopted for this purpose (16,26). Although convolutional deep learning models are powerful to classify and segment images, they lack an intrinsic mechanism to hold information.…”

“…Over the last decade, a range of approaches has been developed to classify behavioral syllables (Kabra et al, 2013;Pereira et al, 2020;Wiltschko et al, 2020;Dunn et al, 2021;Hsu and Yttri, 2021;Segalin et al, 2021;Jia et al, 2022;Luxem et al, 2022;Harris et al, 2023;Luxem et al, 2023;Weinreb et al, 2023). Since these approaches including SaLSa are applied after body parts detection, they can be applied to videos taken in relatively dark environments as long as body parts are detected reliably.…”

“…There are two broad categories of methods used to classify behavioral syllables. The first category is a top-down approach, which involves pre-defining a set of rules to identify behavioral syllables or applying supervised machine learning to classify them (14,15,16).…”

“…There are two broad categories of methods used to classify behavioral syllables. The first category is a top-down approach, which involves pre-defining a set of rules to identify behavioral syllables or applying supervised machine learning to classify them (Kabra et al, 2013;Segalin et al, 2021;Harris et al, 2023). The second category is a bottom-up approach, which involves analyzing data patterns using unsupervised classification algorithms (Dunn et al, 2021;Hsu and Yttri, 2021;Gabriel et al, 2022;Jia et al, 2022;Luxem et al, 2022;Weinreb et al, 2023).…”