3D visualization processes for recreating and studying organismal form

Irschick, Duncan J.; Christiansen, Fredrik; Hammerschlag, Neil; Martín, J. San; Madsen, Peter T.; Wyneken, Jeanette; Brooks, Annabelle; Gleiss, Adrian C.; Fossette, Sabrina; Siler, Cameron D.; Gamble, Tony; Fish, Frank E.; Siebert, Ursula; Patel, Jaymin; Zhan, Xiaowei; Kalogerakis, Evangelos; Medina, Josh; Mukherji, Atreyi; Mandica, Mark; Zotos, Savvas; Detwiler, Jared; Perot, Blair; Lauder, George V.

doi:10.1016/j.isci.2022.104867

Cited by 14 publications

(13 citation statements)

References 95 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we focus on insects, first because of personal predilection, and second because an exoskeleton avoids the need to simulate the complex soft tissue deformation associated with postural changes in animals with endoskeletons. However, powerful approaches to create photo-realistic models of vertebrates exist [53][54][55], and replicAnt is not limited to arthropod models (or even just animals, for what it is worth). The cleaned and rigged model is imported into a pre-configured Unreal Engine 5 project, where a simplified collision mesh is computed to enable interactions with objects inside the simulated world (Fig.…”

Section: Resultsmentioning

confidence: 99%

replicAnt: A pipeline for generating annotated images of animals in complex environments using Unreal Engine

Plum

Bulla²,

Beck

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep learning-based computer vision methods are transforming animal behavioural research. Transfer learning has enabled work in non-model species, but still requires hand-annotation of example footage, and is only performant in well-defined conditions. To overcome these limitations, we created replicAnt, a configurable pipeline implemented in Unreal Engine 5 and Python, designed to generate large and variable training datasets on consumer-grade hardware instead. replicAnt places 3D animal models into complex, procedurally generated environments, from which automatically annotated images can be exported. We demonstrate that synthetic data generated with replicAnt can significantly reduce the hand-annotation required to achieve benchmark performance in common applications such as animal detection, tracking, pose-estimation, and semantic segmentation; and that it increases the subject-specificity and domain-invariance of the trained networks, so conferring robustness. In some applications, replicAnt may even remove the need for hand-annotation altogether. It thus represents a significant step towards porting deep learning-based computer vision tools to the field.

show abstract

Section: Resultsmentioning

confidence: 99%

replicAnt: A pipeline for generating annotated images of animals in complex environments using Unreal Engine

Plum

Bulla²,

Beck

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The field of whole body, three‐dimensional animal modeling is still in its early stages and the many uses of this technology are just starting to be realized. In a recent paper, Irschick et al (2022) identified six primary applications for 3D models of animals: (1) natural history collections, (2) scaling and body condition, (3) robotics, (4) machine learning and locomotion, (5) computational fluid dynamics modeling, and (6) education. For large whales, in particular, three‐dimensional models may open up areas of research that are not otherwise possible.…”

Section: Figurementioning

confidence: 99%

“…

With the proliferation of scanning hardware and photogrammetry software, three-dimensional models of animals have become an increasingly important tool in the study of biology (Adamczak et al, 2019;Irschick et al, 2020Irschick et al, , 2022. Although the field is still emerging, there have already been many creative uses of this technology.

…”

mentioning

confidence: 99%

“…The model can then be directly compared to the moving video footage for a final round of sculpting. The procedure for creating the geometric mesh based on reference images of live animals is described in further detail in Irschick et al (2020Irschick et al ( , 2022.…”

mentioning

confidence: 99%

“…The field of whole body, three-dimensional animal modeling is still in its early stages and the many uses of this technology are just starting to be realized. In a recent paper, Irschick et al (2022) identified six primary applications for 3D models of animals: (1) natural history collections, (2) scaling and body condition,…”

mentioning

confidence: 99%

See 2 more Smart Citations