Machine vision methods for analyzing social interactions

Robie, Alice A.; Seagraves, Kelly M.; Egnor, S.E. Roian; Branson, Kristin

doi:10.1242/jeb.142281

Cited by 129 publications

(110 citation statements)

References 70 publications

Supporting

Mentioning

101

Contrasting

Order By: Relevance

“…This article covers applications of computer vision to find, count and study animals in natural landscapes using images collected in the human visual spectrum. Applications from specimen morphometrics, microscopy (Pennekamp & Schtickzelle, 2013) and animal tracking in laboratory settings are reviewed elsewhere (Dell et al, 2014;Robie, Seagraves, Egnor, & Branson, 2017). To find articles, I used Web of Science to search for "Computer Vision AND (Ecology OR Animals)," yielding 284 articles.…”

Section: Applications Of Computer Vision To Animal Ecologymentioning

confidence: 99%

See 1 more Smart Citation

A computer vision for animal ecology

Weinstein

2017

Journal of Animal Ecology

351

287

View full text Add to dashboard Cite

A central goal of animal ecology is to observe species in the natural world. The cost and challenge of data collection often limit the breadth and scope of ecological study. Ecologists often use image capture to bolster data collection in time and space. However, the ability to process these images remains a bottleneck. Computer vision can greatly increase the efficiency, repeatability and accuracy of image review. Computer vision uses image features, such as colour, shape and texture to infer image content. I provide a brief primer on ecological computer vision to outline its goals, tools and applications to animal ecology. I reviewed 187 existing applications of computer vision and divided articles into ecological description, counting and identity tasks. I discuss recommendations for enhancing the collaboration between ecologists and computer scientists and highlight areas for future growth of automated image analysis.

show abstract

Section: Applications Of Computer Vision To Animal Ecologymentioning

confidence: 99%

“…Three-dimensional imaging has recently been used to track animal behaviour within large indoor enclosures (e.g. Barnard et al, 2016), and applying these tools to animals in natural landscapes is an developing area of research (Robie et al, 2017).…”

Section: Descriptionmentioning

confidence: 99%

A computer vision for animal ecology

Weinstein

2017

Journal of Animal Ecology

351

287

View full text Add to dashboard Cite

show abstract

“…The type of data provided by these barcodes also provides new opportunities for analysis. Using machine learning, it will be possible to automatically classify behaviours and interactions over extended periods of time while also minimizing manual annotation by a human observer (Robie, Seagraves, Egnor, & Branson, ), thereby avoiding bias and fatigue. Such approaches have been developed for studying other organisms (Kabra, Robie, Rivera‐Alba, Branson, & Branson, ), which use data that are similar to what our system generates.…”

Section: Discussionmentioning

confidence: 99%

An automated barcode tracking system for behavioural studies in birds

et al. 2018

View full text Add to dashboard Cite

Recent advances in technology allow researchers to automate the measurement of animal behaviour. These methods have multiple advantages over direct observations and manual data input as they reduce bias related to human perception and fatigue, and deliver more extensive and complete datasets that enhance statistical power. One major challenge that automation can overcome is the observation of many individuals at once, enabling whole‐group or whole‐population tracking. We provide a detailed description of an automated system for tracking birds. Our system uses printed, machine‐readable codes mounted on backpacks. This simple, yet robust, tagging system can be used simultaneously on multiple individuals to provide data on bird identity, position and directionality. Furthermore, because the backpacks are printed on paper, they are very lightweight. We show that our method is reliable, relatively easy to implement and monitor, and with proper handling, has proved to be safe for the birds over long periods of time. We describe the deployment procedure of this system for a captive population of songbirds. We test different camera options, and discuss their advantages and disadvantages. In particular, we highlight how using single‐board computers to control the frequency and duration of image capture makes this system affordable and adaptable to a range of study systems and research questions. The ability to automate the measurement of individual positions has the potential to significantly increase the power of both observational and experimental studies. The system can capture both detailed interactions (using video recordings) and repeated observations (e.g. once per second for the entire day) of individuals over long timescales (months or potentially years). This approach opens the door to tracking life‐long relationships among individuals, while also capturing fine‐scale differences in behaviour.

show abstract

“…38 Yet aquatic ecosystems appear to be poorly represented in movement ecology 39 research with only nine publications in the journal Movement Ecology containing one of 40 the following keywords in the title: 'ocean', 'aquatic', 'littoral', 'marine', 'fish', 'sea' [21]. 41 Moreover, of the research into marine or freshwater animal movement, there is a heavy 42 bias towards larger marine fauna, which often inhabit open oceanic areas [21]. This gap 43 of knowledge comes as a surprise when considering the vast attention coastal marine 44 systems receive in other areas of economic and usage management.…”

Section: Introductionmentioning

confidence: 99%

A low-cost, open-source framework for tracking and behavioural analysis of animals in aquatic ecosystems

Francisco

Nuehrenberg

Jordan

2019

Preprint

View full text Add to dashboard Cite

Although methods for tracking animals underwater exist, they frequently involve costly infrastructure investment, or capture and manipulation of animals to affix or implant tags. These practical concerns limit the taxonomic coverage of aquatic movement ecology studies and implementation in areas where high infrastructure investment is impossible. Here we present a method based on deep-learning and structure-from-motion, with which we can accurately determine the 3D location of animals, the structure of the environment in which they are moving. Further behavioural decomposition of the body position and contour of animals subsequently allow quantifying the behavioural states of each interacting animal. This approach can be used with minimal infrastructure and without confining animals to to a fixed area, or capturing and interfering with them in any way. With this approach, we are able to track single individuals (Conger Eel, Conger oceanus), small heterospecific groups (Mullus surmuletus, Diplodus sp.), and schools of animals (Tanganyikan cichlids Lamprologus callipterus) in freshwater and marine systems, and in habitats ranging in environmental complexity. Positional information was highly accurate, with errors as low as 1.67% of body length. Tracking data was embedded in 3D environmental models that could be used to examine collective decision making, obstacle avoidance, and visual connectivity of groups. By analyzing body contour and position, we were also able to use unsupervised classification to quantify the kinematic behavioural states of each animal. The proposed framework allows us to understand animal behaviour in aquatic systems at an unprecedented resolution and a fraction of the cost of established methodologies, with minimal domain expertise at the data acquisition or analysis phase required. Implementing this method, research can be conducted in a wide range of field contexts to collect laboratory standard data, vastly expanding both the taxonomic and environmental coverage of quantitative animal movement analysis with a low-cost, open-source solution. 1/151 Understanding the movement and behaviour of animals in their natural habitats is the 2 ultimate goal of behavioural and movement ecology. By situating our studies in the 3 natural world, we have the potential to uncover the natural processes of selection acting 4 on the behaviour in natural populations, in a manner that cannot be achieved through 5 lab studies alone. The ongoing advance of animal tracking and biologging has the 6 potential to revolutionize not only the scale of data collected from wild systems, but 7 also the types of questions that can subsequently be answered. Incorporating 8 geographical data has already given insights, for example, into the homing behaviour of 9 reef fish, migratory patterns of birds, or the breeding site specificity of sea 10 turtles [7,17,46]. Great advances in systems biology have further been made through 11 the study of movement ecology, understanding migratory patterns of birds traversing 12 their phys...

show abstract

Machine vision methods for analyzing social interactions

Cited by 129 publications

References 70 publications

A computer vision for animal ecology

A computer vision for animal ecology

An automated barcode tracking system for behavioural studies in birds

A low-cost, open-source framework for tracking and behavioural analysis of animals in aquatic ecosystems

Contact Info

Product

Resources

About