Using deep neural networks to model similarity between visual patterns: Application to fish sexual signals

Hulse, Samuel; Renoult, Julien P.; Mendelson, Tamra C.

doi:10.1016/j.ecoinf.2021.101486

Cited by 7 publications

(5 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The previous classification of drawings according to season based on transfer learning was used as a baseline to compare other classification approaches intended to explore the role of different features in seasonal variation. Here, we focus on the features described by activations [ 20 ]. As shallow layers encode local information represented by simple features, and deeper layers encode global and more complex information, we first compared the ability of these different layers to classify seasons.…”

Section: Methodsmentioning

confidence: 99%

Using Artificial Intelligence to Analyze Non-Human Drawings: A First Step with Orangutan Productions

Beltzung

Péle

Renoult

et al. 2022

Animals

View full text Add to dashboard Cite

Drawings have been widely used as a window to the mind; as such, they can reveal some aspects of the cognitive and emotional worlds of other animals that can produce them. The study of non-human drawings, however, is limited by human perception, which can bias the methodology and interpretation of the results. Artificial intelligence can circumvent this issue by allowing automated, objective selection of features used to analyze drawings. In this study, we use artificial intelligence to investigate seasonal variations in drawings made by Molly, a female orangutan who produced more than 1299 drawings between 2006 and 2011 at the Tama Zoological Park in Japan. We train the VGG19 model to first classify the drawings according to the season in which they are produced. The results show that deep learning is able to identify subtle but significant seasonal variations in Molly’s drawings, with a classification accuracy of 41.6%. We use VGG19 to investigate the features that influence this seasonal variation. We analyze separate features, both simple and complex, related to color and patterning, and to drawing content and style. Content and style classification show maximum performance for moderately complex, highly complex, and holistic features, respectively. We also show that both color and patterning drive seasonal variation, with the latter being more important than the former. This study demonstrates how deep learning can be used to objectively analyze non-figurative drawings and calls for applications to non-primate species and scribbles made by human toddlers.

show abstract

Section: Methodsmentioning

confidence: 99%

Using Artificial Intelligence to Analyze Non-Human Drawings: A First Step with Orangutan Productions

Beltzung

Péle

Renoult

et al. 2022

Animals

View full text Add to dashboard Cite

show abstract

“…Specifically, using style transfer, one can distinguish between two camouflage strategies: background matching, which depends on texture (i.e., patterns and variations in color or shading) to blend in with the environment, and disruptive coloration, which hinders individual recognition by breaking up a visual outline, thereby creating visual confusion. Using a similar approach in fish, Hulse et al (2022) compared the gram matrices of images of male and female darters to habitat images of different categories to determine whether fish species were more visually similar to the habitat in which they occur as compared to habitats in which they are typically not found. Female patterns were found more similar to their habitats than were nuptial male patterns, highlighting a potential trade-off between sexual and natural selection (i.e., camouflage) for males during the mating season.…”

Section: Discussionmentioning

confidence: 99%

Using Neural Style Transfer to study the evolution of animal signal design: A case study in an ornamented fish

Héjja-Brichard

Million

Renoult

et al. 2023

Preprint

View full text Add to dashboard Cite

The sensory drive hypothesis of animal signal evolution suggests that animal communication signals evolve in response to environmental pressures. While classical approaches to testing this hypothesis focus on one aspect of the signal, deep learning techniques like generative models can create and manipulate stimuli without targeting a specific feature. Here, we used a technique called style transfer to experimentally test preferences for colour patterns in a fish. We manipulated how similar or dissimilar male body patterns were to their habitats using the Neural Style Transfer (NST) algorithm. We predicted that males whose body patterns are similar to their habitats are easier to process and thus preferred by conspecifics. Our findings suggest that both males and females tend to be sensitive to habitat congruence in their preferences, but to different extents, requiring additional investigation. Nonetheless, this study demonstrates the potential of deep learning techniques in testing hypotheses about animal communication signals.

show abstract

“…However, the validity of similarity assessments in biological studies, where ground-truth data are seldom available, relies solely on the accuracy of species or subspecies classification [9,14], which may not always reflect similarity. We solved this problem by using the pictorial book's description of the difficulties of discriminating sexes as ground-truth data, which provided stronger evidence for the accuracy of similarity measurements using CNN.…”

Section: The Accuracy Of Cnn-based Analysismentioning

confidence: 99%

“…Similarly, the intermediate output of a CNN pre-trained on a large-scale dataset has been shown to be useful in assessing the similarity between input images, even when CNNs are not trained for those input images [11,12]. Pre-trained CNNs have been shown to measure similarities successfully on biological images, such as bumble bees [13] and fish body texture [14]. The CNN-based similarity measurement appear to solve the difficulty in dorsoventral comparison of butterfly wing patterns.…”

Section: Introductionmentioning

confidence: 99%

Dorsoventral comparison of intraspecific polymorphisms in the butterfly wing pattern using a convolutional neural network

Amino,

Hirakasa,

Yago

et al. 2024

Preprint

View full text Add to dashboard Cite

Butterfly wing patterns exhibit notable differences between the dorsal and ventral surfaces, and morphological analyses of them have provided insights into the ecological and behavioural characteristics of wing colour patterns. Conventional methods for dorsoventral comparisons are constrained by the need for homologous patches or shared features between two surfaces, limiting their applicability across species. We used a convolutional neural network (CNN)-based analysis, which can compare images of the dorsal and ventral surfaces without focusing on homologous patches or features, to detect dorsoventral bias in intraspecific polymorphisms such as sexual dimorphism (SD) and female-limited mimetic polymorphism (FMP). Using specimen images of 29 butterfly species from the Yaeyama Islands, Japan, we first showed that the level of SD calculated by CNN-based analysis corresponded well with traditional assessments of SD, demonstrating the validity of the method. Dorsoventral biases were widely detected, particularly in SD, which tended to bias dorsally. This supports the conventional hypothesis that sexual selection acts more strongly on the dorsal surface. In contrast, the FMP analysis showed no significant bias. Our findings highlight CNN-based analysis as a versatile technique for dorsoventral comparisons, suggesting that broader species sampling could reveal general patterns of selection acting differentially on the two surfaces.

show abstract

Using deep neural networks to model similarity between visual patterns: Application to fish sexual signals

Cited by 7 publications

References 41 publications

Using Artificial Intelligence to Analyze Non-Human Drawings: A First Step with Orangutan Productions

Using Artificial Intelligence to Analyze Non-Human Drawings: A First Step with Orangutan Productions

Using Neural Style Transfer to study the evolution of animal signal design: A case study in an ornamented fish

Dorsoventral comparison of intraspecific polymorphisms in the butterfly wing pattern using a convolutional neural network

Contact Info

Product

Resources

About