Unsupervised behaviour analysis and magnification (uBAM) using deep learning

Brattoli, Biagio; Büchler, Uta; Dorkenwald, Michael; Reiser, Philipp; Filli, Linard; Helmchen, Fritjof; Wahl, Anna-Sophia; Ommer, Björn

doi:10.1038/s42256-021-00326-x

Cited by 24 publications

(21 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We provide four videos (poking_plants_ [1][2][3][4].mp4) for the PokingPlants dataset showing distinct types of plants of substantially different shapes and appearances. Despite these large variances, our model generates realistic and appealing visualizations which are plausible responses to the poke.…”

Section: A1 Pokingplantsmentioning

confidence: 99%

“…For the iPER [49] dataset, we also provide four videos (iper_ [1][2][3][4].mp4) containing unseen actors in indoor as well as outdoor scenes. When comparing the synthesized motion, which is generated based on simulated pokes (second columns), with the ground truth sequences depicted in the first columns, one can clearly observe, that our proposed method achieves to infer realistic global motion only from those sparse, localized interactions.…”

Section: A2 Ipermentioning

confidence: 99%

“…Thus, we use it to demonstrate our method to be also able to handle such real-world conditions. As for the other datasets discussed so far, we prepared four videos (taichi_ [1][2][3][4].mp4; the individual image frames are shown in Figures 14-17). Also for this dataset, the model infers plausible global motions from the poke as indicated by the examples obtained from the simulated poke, which are similar to the ground truth sequences for all provided visualizations.…”

Section: A3 Tai-chi-hdmentioning

confidence: 99%

“…Finally, we illustrate the generalization capabilities of our approach by applying it to four unseen types of plants, which were obtained by using image search in the internet. The resulting generations can be viewed in generalization_ [1][2][3][4].mp4 within the subfolder --B2-Generalization.…”

Section: B3 Generalization To Unseen Types Of Plantsmentioning

confidence: 99%

“…Directly controlling and, even further, interacting with objects on a local level however, so far, is a novel enterprise. Teaching visual systems to understand the complex dynamics of objects both arising by explicit manipulations of individual parts and to predict and analyze the behavior [17,4,5] of the remainder of the object is an exceptionally challenging task. Similarly to a child in the example above, such systems need to know about the natural interrelations of different parts of an object [88].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Understanding Object Dynamics for Interactive Image-to-Video Synthesis

Blattmann¹,

Milbich²,

Dorkenwald³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

What would be the effect of locally poking a static scene? We present an approach that learns naturallylooking global articulations caused by a local manipulation at a pixel level. Training requires only videos of moving objects but no information of the underlying manipulation of the physical scene. Our generative model learns to infer natural object dynamics as a response to user interaction and learns about the interrelations between different object body regions. Given a static image of an object and a local poking of a pixel, the approach then predicts how the object would deform over time. In contrast to existing work on video prediction, we do not synthesize arbitrary realistic videos but enable local interactive control of the deformation. Our model is not restricted to particular object categories and can transfer dynamics onto novel unseen object instances. Extensive experiments on diverse objects demonstrate the effectiveness of our approach compared to common video prediction frameworks. Project page is available at https://bit.ly/3cxfA2L.

show abstract

Section: A1 Pokingplantsmentioning

confidence: 99%

Section: A2 Ipermentioning

confidence: 99%

Section: A3 Tai-chi-hdmentioning

confidence: 99%

Section: B3 Generalization To Unseen Types Of Plantsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Understanding Object Dynamics for Interactive Image-to-Video Synthesis

Blattmann¹,

Milbich²,

Dorkenwald³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Learning-based Axial Video Motion Magnification

Byung-Ki,

Hyun-Bin,

Jun-Seong

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

SUBTLE: An Unsupervised Platform with Temporal Link Embedding that Maps Animal Behavior

Kwon,

Kim,

Kim

et al. 2024

Int J Comput Vis

View full text Add to dashboard Cite

While huge strides have recently been made in language-based machine learning, the ability of artificial systems to comprehend the sequences that comprise animal behavior has been lagging behind. In contrast, humans instinctively recognize behaviors by finding similarities in behavioral sequences. Here, we develop an unsupervised behavior-mapping framework, SUBTLE (spectrogram-UMAP-based temporal-link embedding), to capture comparable behavioral repertoires from 3D action skeletons. To find the best embedding method, we devise a temporal proximity index (TPI) as a new metric to gauge temporal representation in the behavioral embedding space. The method achieves the best TPI score compared to current embedding strategies. Its spectrogram-based UMAP clustering not only identifies subtle inter-group differences but also matches human-annotated labels. SUBTLE framework automates the tasks of both identifying behavioral repertoires like walking, grooming, standing, and rearing, and profiling individual behavior signatures like subtle inter-group differences by age. SUBTLE highlights the importance of temporal representation in the behavioral embedding space for human-like behavioral categorization.

show abstract

Unsupervised behaviour analysis and magnification (uBAM) using deep learning

Cited by 24 publications

References 50 publications

Understanding Object Dynamics for Interactive Image-to-Video Synthesis

Understanding Object Dynamics for Interactive Image-to-Video Synthesis

Learning-based Axial Video Motion Magnification

SUBTLE: An Unsupervised Platform with Temporal Link Embedding that Maps Animal Behavior

Contact Info

Product

Resources

About