Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals

Liu, Siyuan; Günel, Semih; Ostrek, Mirela; Ramdya, Pavan; Fua, Pascal; Rhodin, Helge

doi:10.1109/cvpr42600.2020.01317

Cited by 38 publications

(38 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although here we optimized for speed and static stability during tethered locomotion, NeuroMechFly can also locomote without body support, allowing one to optimize neural networks for free behaviors. Finally, our model can also be used to create animations of behaviors that improve the training of 3D pose estimation networks for applications in computer vision [46,48].…”

Section: Discussionmentioning

confidence: 99%

“…CC-BY-NC 4.0 International license made available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprint this version posted April 18, 2021. ; https://doi.org/10.1101/2021.04.17.440214 doi: bioRxiv preprint of adult Drosophila that could then be used for neuromechanical studies as well as for model-based computer vision methods [45][46][47][48][49].…”

Section: Constructing a Data-driven Biomechanical Model Of Adult Drosophilamentioning

confidence: 99%

See 1 more Smart Citation

NeuroMechFly, a neuromechanical model of adultDrosophila melanogaster

Rios

Ozdil

Ramalingasetty

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Animal behavior emerges from a seamless interaction between musculoskeletal elements, neural network dynamics, and the environment. Accessing and understanding the interplay between these intertwined systems requires the development of integrative neuromechanical simulations. Until now, there has been no such simulation framework for the widely studied model organism, Drosophila melanogaster. Here we present NeuroMechFly, a data-driven computational model of an adult female fly that is designed to synthesize rapidly growing experimental datasets and to test theories of neuromechanical behavioral control. NeuroMechFly combines a set of modules including an exoskeleton with articulating body parts---limbs, halteres, wings, abdominal segments, head, proboscis, and antennae---muscle models, and neural networks within a physics-based simulation environment. Using this computational framework, we (i) predict the minimal limb degrees-of-freedom needed for real Drosophila behaviors, (ii) estimate expected contact reaction forces, torques, and tactile signals during replayed Drosophila walking and grooming, and (iii) discover neural network and muscle parameters that can drive tripod walking. Thus, NeuroMechFly is a powerful testbed for building an understanding of how behaviors emerge from interactions between complex neuromechanical systems and their physical surroundings.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Constructing a Data-driven Biomechanical Model Of Adult Drosophilamentioning

confidence: 99%

NeuroMechFly, a neuromechanical model of adultDrosophila melanogaster

Rios

Ozdil

Ramalingasetty

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In most cases, it is used as an intermediate phase that adapts the input images to tackle problems with greater efficiency. Among some of the most frequent problems in which this type of network is used, are the classification task [1][2][3][4], the object detection task [5][6][7][8] or the correspondence task [9,10].…”

Section: Introductionmentioning

confidence: 99%

A new training strategy for spatial transform networks (STN’s)

Moya

Puchalt

Castro

et al. 2022

Neural Comput & Applic

View full text Add to dashboard Cite

Spatial transform networks (STN) are widely used since they can transform images captured from different viewpoints to obtain an objective image. These networks use an image captured from any viewpoint as input and the desired image as a label. Usually, these images are segmented, but this could lead to convergence problems if the percentage of overlap between the segmented images is quite low. In this paper, we propose a new training method to facilitate the convergence of a STN in these cases, even when there is no overlap between the object’s projections in the two images. This new strategy is based on the incorporation of the distance transformation images to the training, thus increasing the useful image information to provide gradients in the loss function. This new training strategy has been applied to a real case, with images of Caenorhabditis elegans, and to a simulated case, which uses artificial images to ensure that there is no overlap between the images used for the assays. In the assays carried out with these datasets, we have shown that the training convergence is strengthened, reaching a precision level for IoU metric of 0.862 and 0.984, respectively, and the computational cost has been maintained compared to the assay with segmented images, for the real case.

show abstract

“…A method that classifies different strains of C. elegans using convolutional neural networks (CNN) was presented in [ 17 ]. Methods based on neural networks have also been proposed for head and tail localisation [ 18 ] and pose estimation [ 19 , 20 , 21 ]. Recently, [ 22 , 23 ] used different convolutional neural network models to estimate the physiological age of C. elegans .…”

Section: Introductionmentioning

confidence: 99%

Towards Lifespan Automation for Caenorhabditis elegans Based on Deep Learning: Analysing Convolutional and Recurrent Neural Networks for Dead or Live Classification

Garví

Puchalt

Castro

et al. 2021

Sensors

View full text Add to dashboard Cite

The automation of lifespan assays with C. elegans in standard Petri dishes is a challenging problem because there are several problems hindering detection such as occlusions at the plate edges, dirt accumulation, and worm aggregations. Moreover, determining whether a worm is alive or dead can be complex as they barely move during the last few days of their lives. This paper proposes a method combining traditional computer vision techniques with a live/dead C. elegans classifier based on convolutional and recurrent neural networks from low-resolution image sequences. In addition to proposing a new method to automate lifespan, the use of data augmentation techniques is proposed to train the network in the absence of large numbers of samples. The proposed method achieved small error rates (3.54% ± 1.30% per plate) with respect to the manual curve, demonstrating its feasibility.

show abstract

Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals

Cited by 38 publications

References 35 publications

NeuroMechFly, a neuromechanical model of adultDrosophila melanogaster

NeuroMechFly, a neuromechanical model of adultDrosophila melanogaster

A new training strategy for spatial transform networks (STN’s)

Towards Lifespan Automation for Caenorhabditis elegans Based on Deep Learning: Analysing Convolutional and Recurrent Neural Networks for Dead or Live Classification

Contact Info

Product

Resources

About