Scene grammar shapes the way we interact with objects, strengthens memories, and speeds search

Abstract

Predictions of environmental rules (here referred to as “scene grammar”) can come in different forms: seeing a toilet in a living room would violate semantic predictions, while finding a toilet brush next to the toothpaste would violate syntactic predictions. The existence of such predictions has usually been investigated by showing observers images containing such grammatical violations. Conversely, the generative process of creating an environment according to one’s scene grammar and its effects on behavior and memory has received little attention. In a virtual reality paradigm, we either instructed participants to arrange objects according to their scene grammar or against it. Subsequently, participants’ memory for the arrangements was probed using a surprise recall (Exp1), or repeated search (Exp2) task. As a result, participants’ construction behavior showed strategic use of larger, static objects to anchor the location of smaller objects which are generally the goals of everyday actions. Further analysis of this scene construction data revealed possible commonalities between the rules governing word usage in language and object usage in naturalistic environments. Taken together, we revealed some of the building blocks of scene grammar necessary for efficient behavior, which differentially influence how we interact with objects and what we remember about scenes.

Disrupted object-scene semantics boost scene recall but diminish object recall in drawings from memory

“…These results help to reconcile a debate in the field on the impact of object-scene semantics on memory (Friedman, 1979;Pedzek et al, 1979;Hollingworth et al, 2001;Draschkow & Võ, 2017;Cornelissen & Võ, 2017). Using drawing as a memory output allows for a fine-grained look at how object-scene semantics influence memory representations, and we observe a nuanced trade-off in which memory for the overall image is better, but memory for the objects within it is worse.…”

“…One possibility is that inconsistent objects are distinctive and easier to remember (Friedman, 1979;Pedzek, Whetstone, Reynolds, Askari, & Dougherty, 1989;Hollingworth, Williams & Henderson, 2001). Conversely, we might instead find that consistent objects better fit our scene schemas and thus are easier to remember, as has been found in recent work analyzing the role of consistency on scene construction (Draschkow & Võ, 2017). A third possibility is that we may observe no memory difference between inconsistent and consistent objects (Cornelissen & Võ, 2017).…”

Extra-foveal Processing of Object Semantics Guides Early Overt Attention During Visual Search

“…For example, participants searched for a target (e.g., a banana) in an object array comprising a semantic competitor (e.g., a monkey), a visually similar competitor (e.g., a canoe) and two more unrelated distractors (e.g., tambourine and hat). A seminal study by Duncan and Humphreys (1989) showed that when distractors differ homogenously from the target on a target-defining visual feature (e.g., the colour), the guidance of such feature on directing overt attention is very strong. When distractors are instead more heterogeneous, then such a feature has a weaker effect.…”

Typical retinotopic locations impact the time course of object coding

“…More similarly to our study, others have investigated the effects of syntactic consistencies (i.e., whether the object is placed in its typical within-scene location), and found comparably late effects, most prominently between 400 and 600 ms (Demiral, Malcolm, & Henderson, 2012; Võ & Wolfe, 2013). Such effects of syntactic consistencies have been linked to efficient behavioral performance in complex multi-object scenes (Draschkow & Võ, 2017).…”