Cross-form facilitation effects from simultaneous gesture/word combinations with ERP analysis

Abstract

The aim of this study was to investigate if simultaneous redundant forms of stimuli involving gesture and word combinations would produce a facilitation effect in reaction time as well as produce an neurological semantic incongruity effect called an N400. Event-related potentials (ERPs) were recorded from 14 scalp electrode sites while subject performed reactiontime decision tasks. For each trial, subjects had to indicate if the single form or cross-form stimuli were oriented upwards or downwards, or if they contradicted each other (only applicable in cross-form stimuli were there were two stimuli). The uniform stimuli were defined by a picture of a man pointing up or down, a picture of just a hand pointing up or down, or various word combinations such as up/down, high/low, above/below, and top/bottom. The cross-form stimuli were combinations of the words and pictures. The results show that while there was no N400 effect, there was however an facilitation effect regarding the positioning of the cross-form stimuli indicating that people have a tendency to pay more attention to gestures than words for information. A. IntroductionUnimodality studies (e.g. two visual or two auditory stimuli) have been used in many semantic facilitation studies investigating if several types of stimuli can facilitate identical/synonymous subsequent stimuli of the same modality. These can be either within-form (e.g. word-word or picturepicture) or cross-form (e.g. picture-word or word-picture) and some examples of them include [1], [2] who found that the results were highly dependent on the task requirement. Name verification, where subjects had to determine if a subsequent stimulus had the same name as the previous one, facilitated picture targets. Category verification, where subjects had to determine if a subsequent stimulus was in the same category as the previous one, facilitated both pictures and words [2].Redundancy gain, also referred to as the redundant-signal effect, is a common phenomenon found in behavioral studies where performance is enhanced with the presentation of simultaneous stimuli requiring the same response compared to either stimulus alone. The redundancy gain can enhance reaction time [3], [4], [5], [6], increase accuracy [7], [8], [9], [10], as well as increase response forcefulness [11], [12].Redundancy gain can reflect probability summation (also known as the race model or separate-activations model), [4], [13] where the redundant stimuli (e.g. the response to same target on left, right, or simultaneous visual fields) are presented and the one detected first produces the fastest response. Since the channel processed first produces the fastest response, statistical facilitation has been achieved. The second way to explain redundancy gain is by coactivation [14], [15] where reaction to stimuli defined by different features (e.g. shape/color) or modalities (e.g. audio/visual) occurs and the coactivation of both is combined to produce a naturally faster response than either alone [13]. The majority of redundancy ...