Material category of visual objects computed from specular image structure

Schmid, Alexandra C.; Barla, Pascal; Doerschner, Katja

doi:10.1038/s41562-023-01601-0

Cited by 8 publications

(3 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Then, we assign the same label y 1 across different illuminations. Recent literature shows that more complex visual features are better correlated to human perception [SBD23] but these require access to matte and specular components separately, so we choose skewness for its simplicity.…”

Section: Automatic Weak Labelsmentioning

confidence: 99%

Predicting Perceived Gloss: Do Weak Labels Suffice?

Guerrero‐Viu,

Subias,

Serrano

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

Estimating perceptual attributes of materials directly from images is a challenging task due to their complex, not fully‐understood interactions with external factors, such as geometry and lighting. Supervised deep learning models have recently been shown to outperform traditional approaches, but rely on large datasets of human‐annotated images for accurate perception predictions. Obtaining reliable annotations is a costly endeavor, aggravated by the limited ability of these models to generalise to different aspects of appearance. In this work, we show how a much smaller set of human annotations (“strong labels”) can be effectively augmented with automatically derived “weak labels” in the context of learning a low‐dimensional image‐computable gloss metric. We evaluate three alternative weak labels for predicting human gloss perception from limited annotated data. Incorporating weak labels enhances our gloss prediction beyond the current state of the art. Moreover, it enables a substantial reduction in human annotation costs without sacrificing accuracy, whether working with rendered images or real photographs.

show abstract

Section: Automatic Weak Labelsmentioning

confidence: 99%

Predicting Perceived Gloss: Do Weak Labels Suffice?

Guerrero‐Viu,

Subias,

Serrano

et al. 2024

Computer Graphics Forum

View full text Add to dashboard Cite

show abstract

“…Based on visual input, we can often distinguish materials, and infer their diverse optical properties (e.g., surface glossiness [16][17][18][19] , translucency [20][21][22][23][24][25][26][27][28][29] or transparency 30 ), surface properties (e.g., roughness 31 ), mechanical properties (e.g., softness 32 , stiffness 33 ) and states (e.g., freshness 34 , wetness 35 ). Previous works actively examined how visual estimates of material attributes are related to the statistical image features 36 , as well as seeking to probe the neural representation of material perception in cortical areas of the ventral visual pathway [37][38][39][40] .…”

Section: Introductionmentioning

confidence: 99%

“…Verbal description could serve as an interpretable representation that encodes the salient features of material qualities. While a plethora of works scrutinized the visual estimation of specific material properties related to physics 17,36,[41][42][43] , few studies shined the light on more subjective material perception from both visual judgment and language expression. With a large-scale image dataset of materials, Schmidt et al (2023) 44 used visual triplet similarity judgments from crowd-sourcing to distill a representational space, which was later annotated by humans to find conceptual and perceptual dimensions of materials.…”

Section: Introductionmentioning

confidence: 99%

Probing the Link Between Vision and Language in Material Perception Using Psychophysics and Unsupervised Learning

Liao,

Sawayama,

Xiao

2024

Preprint

View full text Add to dashboard Cite

The ability to visually discriminate and recognize materials (e.g., candy or crystal) is crucial in planning actions (e.g., determining edibility) in the environment. Meanwhile, language is a powerful channel for communicating the characteristics of materials. However, the connection between these two modalities in material perception has remained elusive. Here, we created a diverse set of perceptually convincing material appearances with deep generative networks that model the statistical structure of real-world photos. Besides generating familiar materials (e.g., soaps, rocks, and squishy toys), we also synthesized unfamiliar novel materials with cross-category morphing (e.g., transforming a soap into a rock). With stimuli sampled from this expansive space, we compared the representations of materials from two cognitive tasks: visual material similarity judgments and verbal descriptions. Our analysis unveiled a moderate but significant correlation between vision and language within individuals. In contrast, the image-based representation derived from the latent code of the generative model only weakly correlated with human visual judgments. Furthermore, we showed that joining image- and semantic-level representations could improve the prediction of human perception. Material perception may actively involve a high-level understanding of materials and objects to resolve the ambiguity of visual information, and it cannot be merely explained by low-to-mid-level image features. Together, our findings highlight the necessity of leveraging semantic description and visual feature extraction to unveil the critical dimensions of material perception and further improve computer vision models targeting material-related tasks.

show abstract