Triple-stream Deep Metric Learning of Great Ape Behavioural Actions

Brookes, Otto; Mirmehdi, Majid; Kühl, Hjalmar S.; Burghardt, Tilo

doi:10.48550/arxiv.2301.02642

Cited by 2 publications

(2 citation statements)

References 38 publications

(51 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Machine learning applications for camera trap imagery Applications of machine learning to camera trap data mainly focused on animal tracking (21,22) and species recognition (23,24). During the last decade, the development of convolutional neural networks (CNNs) largely improved the performance of vision models for animal detection (7,22,25,26), species classification (17,(27)(28)(29)(30), behavior recognition (8,31) or animal counting (8,32). In 2018, Norouzzadeh et al (8) showed an innovative pipeline to classify species, count animals and assess age, behavior, and interactions with other individuals from the Snapshot Serengeti consensus data, still making it one of the most diverse multilabel classification method for camera traps to date.…”

Section: Background and Related Workmentioning

confidence: 99%

WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models

Gabeff,

Rußwurm,

Tuia

et al. 2023

Preprint

View full text Add to dashboard Cite

Wildlife observation with camera traps has great potential for ethology and ecology, as it gathers data non-invasively in an automated way. However, camera traps produce large amounts of uncurated data, which is time-consuming to annotate. Existing methods to label these data automatically commonly use a fixed pre-defined set of distinctive classes and require many labeled examples per class to be trained. Moreover, the attributes of interest are sometimes rare and difficult to find in large data collections. Large pretrained vision-language models, such as Contrastive Language Image Pretraining (CLIP), offer great promises to facilitate the annotation process of camera-trap data. Images can be described with greater detail, the set of classes is not fixed and can be extensible on demand and pretrained models can help to retrieve rare samples. In this work, we explore the potential of CLIP to retrieve images according to environmental and ecological attributes. We create WildCLIP by fine-tuning CLIP on wildlife camera-trap images and to further increase its flexibility, we add an adapter module to better expand to novel attributes in a few-shot manner. We quantify WildCLIP’s performance and show that it can retrieve novel attributes in the Snapshot Serengeti dataset. Our findings outline new opportunities to facilitate annotation processes with complex and multi-attribute captions. The code will be made available athttps://github.com/amathislab/wildclip.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models

Gabeff,

Rußwurm,

Tuia

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The authors also propose an action recognition network, CARe, that recognizes the actions of unseen animal species. In the paper [20], the authors introduce a metric learning system with a triple-stream embedding network using RGB and optical flow features for the action recognition of great apes.…”

Section: Introductionmentioning

confidence: 99%

Action Detection for Wildlife Monitoring with Camera Traps Based on Segmentation with Filtering of Tracklets (SWIFT) and Mask-Guided Action Recognition (MAROON)

Schindler,

Steinhage,

van Beeck Calkoen

et al. 2024

Applied Sciences

View full text Add to dashboard Cite

Behavioral analysis of animals in the wild plays an important role for ecological research and conservation and has been mostly performed by researchers. We introduce an action detection approach that automates this process by detecting animals and performing action recognition on the detected animals in camera trap videos. Our action detection approach is based on SWIFT (segmentation with filtering of tracklets), which we have already shown to successfully detect and track animals in wildlife videos, and MAROON (mask-guided action recognition), an action recognition network that we are introducing here. The basic ideas of MAROON are the exploitation of the instance masks detected by SWIFT and a triple-stream network. The instance masks enable more accurate action recognition, especially if multiple animals appear in a video at the same time. The triple-stream approach extracts features for the motion and appearance of the animal. We evaluate the quality of our action recognition on two self-generated datasets, from an animal enclosure and from the wild. These datasets contain videos of red deer, fallow deer and roe deer, recorded both during the day and night. MAROON improves the action recognition accuracy compared to other state-of-the-art approaches by an average of 10 percentage points on all analyzed datasets and achieves an accuracy of 69.16% on the Rolandseck Daylight dataset, in which 11 different action classes occur. Our action detection system makes it possible todrasticallyreduce the manual work of ecologists and at the same time gain new insights through standardized results.

show abstract

Triple-stream Deep Metric Learning of Great Ape Behavioural Actions

Cited by 2 publications

References 38 publications

WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models

WildCLIP: Scene and animal attribute retrieval from camera trap data with domain-adapted vision-language models

Action Detection for Wildlife Monitoring with Camera Traps Based on Segmentation with Filtering of Tracklets (SWIFT) and Mask-Guided Action Recognition (MAROON)

Contact Info

Product

Resources

About