No abstract

A variety of simple models has been proposed to understand the collective motion of animals. These models can be insightful but may lack important elements necessary to predict the motion of each individual in the collective. Adding more detail increases predictability but can make models too complex to be insightful. Here we report that deep attention networks can obtain a model of collective behavior that is simultaneously predictive and insightful thanks to an organization in modules. When using simulated trajectories, the model recovers the ground-truth interaction rule used to generate them, as well as the number of interacting neighbours. For experimental trajectories of large groups of 60-100 zebrafish, Danio rerio, the model obtains that interactions between pairs can approximately be described as repulsive, attractive or as alignment, but only when moving slowly. At high velocities, interactions correspond only to alignment or alignment mixed with repulsion at close distances. The model also shows that each zebrafish decides where to move by aggregating information from the group as a weighted average over neighbours. Weights are higher for neighbours that are close, in a collision path or moving faster in frontal and lateral locations. The network also extracts that the number of interacting individuals is dynamical and typically in the range 8–22, with 1–10 more important ones. Our results suggest that each animal decides by dynamically selecting information from the collective.

Our understanding of collective animal behavior is limited by our ability to track each of the individuals. We describe an algorithm and software, idtracker.ai, that extracts from video all trajectories with correct identities at a high accuracy for collectives of up to 100 individuals. It uses two deep networks, one detecting when animals touch or cross and another one for animal identi cation, trained adaptively to conditions and di culty of the video.Obtaining animal trajectories from a video faces the problem of how to track animals with correct identities after they touch, cross or they are occluded by environmental features. To bypass this problem, we proposed in idTracker the idea of tracking by identi cation of each individual using a set of reference images obtained from the video [1]. idTracker and further developments in animal identi cation algorithms [2-6] can work for small groups of 2-15 individuals. In larger groups, they only work for particular videos with few animal crossings [7] or with few crossings of particular species-speci c features [5].Here we present idtracker.ai, a system to track all individuals in small or large collectives (up to 100 individuals) at a high identi cation accuracy, often of > 99.9%. The method is species-agnostic and we have tested it in small and large collectives of zebra sh, Danio rerio and ies, Drosophila melanogaster. Code, quickstart guide and data used are provided (see Methods), and Supplementary Text describes algorithms and gives pseudocode. A graphical user interface walks users through tracking, exploration and validation (Fig. 1a).Similar to idTracker [1], but with di erent algorithms, idtracker.ai identi es animals using their visual features. In idtracker.ai, animal identi cation is done adapting deep learning [8][9][10] to work in videos of animal collectives thanks to speci c training protocols. In brief, it consists of a series of processing steps summarized in Fig. 1b. After image preprocessing, the rst deep network nds when animals are touching or crossing. Then the system uses the images between these detected to train a second deep network for animal identi cation. The system rst assumes that a single portion of video when animals do not touch or cross has enough images to properly train the identi cation network (Protocol 1). However, animals touch or cross often and this portion is then typically very short, making the system estimate that identi cation quality is too low. If this happens, two extra 1 . CC-BY-NC 4.0 International license not peer-reviewed) is the author/funder. It is made available under a

A variety of simple models has been proposed to understand the collective motion of animals. These models can be insightful but lack important elements necessary to predict the motion of each individual in the collective. Adding more detail increases predictability but can make models too complex to be insightful. Here we report that deep attention networks can obtain in a data-driven way a model of collective behavior that is simultaneously predictive and insightful thanks to an organization in modules. The model obtains that interactions between two zebrafish, Danio rerio, in a large groups of 60-100, can be approximately be described as repulsive, attractive or as alignment, but only when moving slowly. At high velocities, interactions correspond only to alignment or alignment mixed with repulsion at close distances. The model also shows that each zebrafish decides where to move by aggregating information from the group as a weighted average over neighbours. Weights are higher for neighbours that are close, in a collision path or moving faster in frontal and lateral locations. These weights effectively select 5 relevant neighbours on average, but this number is dynamical, changing between a single neighbour to up to 12, often in less than a second. Our results suggest that each animal in a group decides by dynamically selecting information from the group. Highlights• At 30 days postfertilization, zebrafish, Danio rerio, can move in very cohesive and predictable large groups• Deep attention networks obtain a predictive and understadable model of collective motion• When moving slowly, interations between pairs of zebrafish have clear components of repulsion, attraction and alignment• When moving fast, interactions correspond to alignment and a mixture of alignment and repulsion at close distances • Zebrafish turn left or right depending on a weighted average of interaction information with other fish, with weights higher for close fish, those in a collision path or those moving fast in front or to the sides • Aggregation is dynamical, oscillating between 1 and 12 neighbouring fish, with 5 on average

Methods Software availabilityidtracker.ai is open-source and free software (license GPL v.3). The source-code as well as the instructions for its installation are available in www.gitlab.com/polavieja_lab/idtrackerai. A quick-start user guide and a detailed explanation of the graphical user interface can be found in www.idtracker.ai.