Picture-word differences in a sentence verification task

Abstract

Effects of picture-word format were investigated with four problem-solving items. In Experiment I, picture-word input was presented for 8 sec followed by a test sentence that included verbatim and inference statements. Subjects made a true/false reaction time to the test sentence. In Experiment 2, the input remained on the screen while the test sentence was presented with varied stimulus onset asynchronies from 0 to 1,000msec. Results showed that responses to pictures were faster than responses to words, and the format effect was larger with inference than with verbatim sentences. The picture advantage seemed to be due to the nature of the input and how information is extracted from it. The findings are discussed within the context of text-processing theories (Glenberg & Langston, 1992;Larkin & Simon, 1987).Although format effects have been identified in previous research (Goolkasian & Park, 1980;Kroll & Corrigan, 1981;Pellegrino, Rosinski, Chiesi, & Siegel, 1977; Potter & Falconer, 1975; Smith & Magee, 1980), the results have varied with the kind of task, and there is a lingering debate regarding the nature of the representations that are developed from picture and word versions of stimuli. Dual-coding theory (Paivio, 1971(Paivio, , 1975(Paivio, , 1978 proposed separate but interconnected verbal and symbolic systems in which pictures have some processing advantage because they generate representations in both memory systems, whereas words have only verbal representations. An alternative multimodal theory (Potter, 1979;Seymour, 1973;Snodgrass, 1984) proposed that, in addition to the verbal and symbolic representation, there is a third propositional memory system in which concepts are amodal and equally accessible from both words and pictures. Hypothesizing from a current version ofthis theory, Theios and Amrhein (1989b) have suggested that format differences occur when naming responses are required, because picture naming involves two additional processes not present in naming words: determining the meaning of the picture, and finding a name for the picture. Whenever the task requires conceptual processing, format effects disappear because of an amodal abstract conceptual processor. These experiments extend the previous work with picture-word processing by testing whether format effects occur with problem-solving items that require some memory for background inputs but also require reasoning.This work was supported in part by funds from the Foundation of The University of North Carolina at Charlotte and from the State of North Carolina. Some of these findings were reported at the annual meeting of the Psychonomic Society in St. Louis, November 1994. The author thanks Holly Green and Helen Summer Eubanks for their assistance with this research. Correspondence should be sent to P. Goolkasian, Department of Psychology, UNCC, 9201 University City Boulevard, Charlotte, NC 28223-0001 (e-mail: fjJyOOpag@unccvm.uncc.edu). Clark and Chase (1972) were the first to use the sentencepicture verification task to study concep...

The effectiveness of control structure diagrams in source code comprehension activities

“…Numerous studies indicate that semantic analysis is performed faster for pictures than for text and that graphical information is more easily and efficiently remembered than textual information [19], [43]. These studies suggest that graphical representations of software are inherently useful, though particular representations may not be.…”

Comprehending how visual context influences incremental sentence processing: Insights from ERPs and picture‐sentence verification

“…In addition, a few failures to find the typical congruence effect (longer response latencies for a picture-sentence mismatch than match) with serial picture-sentence presentation have led to concerns about the generality of the paradigm. Despite its occasional use to study language comprehension (e.g., Goolkasian, 1996; Reichle, Carpenter, and Just, 2000; Singer, 2006; Underwood et al, 2004), then, the fact remains that insights obtained with this paradigm have had minimal impact on psycholinguistic theories of online sentence comprehension or of situated sentence comprehension.…”

TraVLR: Now You See It, Now You Don’t! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning

“…Inspired by the word/picture sentence verification task from psycholinguistics (Goolkasian, 1996), we further propose various novel evaluation settings by representing the scene bimodally as both an image and a caption. First, TRAVLR supports the novel cross-modal transfer setting (Figure 1): If pretrained V+L models have learnt a truly multimodal representation, they should be able to learn a reasoning task with input from one modality and perform inference using input from the other modality with little to no extra training.…”