The perceptual buildup of three-dimensional structure from motion

The dynamics of the visual system in combining multiple depth cues were investigated by measuring the temporal change in the perceived 3-D shape of a random-dot stimulus with conflicting kinetic depth effect (KDE) and binocular stereopsis cues. The KDE shape perception dominated for the first few seconds, and then was gradually supplanted by the stereo shape perception. The effects of various pre-adaptation stimuli suggested that the temporal change in the perceived shape resulted from a self-adaptation of the KDE mechanism that occurs mainly at the levels of motion and relative motion detection.There are many cues for depth perception, such as binocular stereopsis, object motion (kinetic depth effect, or KDE), observer's motion (motion parallax), shading, and texture gradient. Our visual system recovers the 3-D structures of the outside world by combining depth estimations from these cues. To investigate how various depth cues are combined, several researchers have used stimuli that involve conflicting depth cues at the same position (Biilthoff & Mallot, 1990;Landy, Maloney, & Young, 1991;Maloney & Landy, 1989;Rogers & Collett, 1989). For example, Maloney and his colleagues (Landy et aI., 1991;Maloney & Landy, 1989) examined depth perception with conflicting KDE and texture gradient cues and postulated that the final depth perception from multiple cues is based on a linear summation of the depths estimated from each cue, unless those estimates are strongly inconsistent. The coefficient of each cue is based on the cue's reliability, as derived from its consistency with the other cues.These studies implicitly assume stable depth perception for stimuli with conflicting depth cues. However, after continuous observation of a display showing conflicting KDE and stereo cues, we noticed that the depth perception changes over time, even though the stimuli do not change. At first, the depth perception determined by the KDE cue dominated, but then the depth perception determined by the stereo cue gradually appeared. We felt All of the experiments reported here were conducted at the ATR Auditory and Visual Perception Research Laboratories. The authors are grateful to the laboratory's president, Eiji Yodogawa, and the department head, Keiichi Ueno, for the opportunity to conduct this research. We also thank Takao Sato (NTT), Farley Norman (Ohio State University), and Michael Landy (New York University), who gave us helpful advice. Address correspondence to K. Uomori, Central Research Laboratories, Matsushita Electric Industrial Co. Ltd., 3-4 Hikari-dai, Seikacho, Soraku-gun Kyoto 619-02, Japan. that clarification of the temporal sequence of this depth change was important in obtaining a proper estimate of the contribution of each cue to the final depth perception. We also expected that the examination of this phenomenon might provide new insights into the dynamics of the stereopsis system, the KDE system, or a system combining them.In the present paper we examine the temporal change in depth perception for a conflicting-cu...

Section: Discussioncontrasting

confidence: 56%

The dynamics of the visual system in combining conflicting KDE and binocular stereopsis cues

Uomori¹,

Nishida

1994

“…One possibility is that the observers were recognizing visible speakers by extracting general facialstructure-from-motion information (e.g., Braunstein, Hoffman, & Pollick, 1990;Christie & Bruce, 1998;Hildreth, Grzywacz, Adelson, & Inada, 1990;Pollick, 1997) and then matching it to the heard voice that best fit that facial structure. Although the results reported above are consistent with this interpretation, a post hoc analysis revealed results less consistent with a structure-from-motion basis.…”

Section: Informational Support For Cross-modal Matchesmentioning

confidence: 99%

Hearing a face: Cross-modal speaker matching using isolated visible speech

Rosenblum

Smith

Nichols

et al. 2006

An experiment was performed to test whether cross-modal speaker matches could be made using isolated visible speech movement information. Visible speech movements were isolated using a pointlight technique. In five conditions, subjects were asked to match a voice to one of two (unimodal) speaking point-light faces on the basis of speaker identity. Two of these conditions were designed to maintain the idiosyncratic speech dynamics of the speakers, whereas three of the conditions deleted or distorted the dynamics in various ways. Some of these conditions also equated video frames across dynamically correct and distorted movements. The results revealed generally better matching performance in the conditions that maintained the correct speech dynamics than in those conditions that did not, despite containing exactly the same video frames. The results suggest that visible speech movements themselves can support cross-modal speaker matching.

“…Any realistic model of the visual system should incorporate a tolerance analysis as a complete description of the stimulus. By way of example we apply the tolerance analysis to the stimuli used in two widely known experiments in which different properties of structure were tested-that is, perceived nonrigidity (Norman & Todd, 1993) and ordering in depth (Hildreth, Grzywacz, Adelson, & Inada, 1990). The analysis explains qualitatively the results of these experiments, illustrating that the results are to a large extent due to stimulus limitations rather than to mechanistic properties of the visual system.…”

mentioning

confidence: 86%

“…This means that the deviations introduced by using orthographic projection are larger than those introduced by stretching along the line of sight. Hildreth et al (1990) provided evidence for the plausibility of the incremental rigidity scheme proposed by Ullman (1984) as a model for the visual system. In this scheme an "internal model" of the 3-D structure of the object is maintained.…”

Section: Perspective Projectionmentioning

confidence: 99%

See 1 more Smart Citation

Structure from motion: A tolerance analysis

Hogervorst

Kappers

Koenderink

1996

For a proper evaluation of the performance of the visual system one needs to analyze the stimuli. From such an analysis it should become clear whether the information required for a certain task is available and to what extent. If we do not perform such an analysis we cannot decide whether the limits reached by the visual system are set by stimulus limitations or by limits of the visual system. Koenderink and van Doorn (1987) performed a tolerance analysis to measure the extent to which certain information is available in velocity flow fields. Here we will do the same for an important class of structure-from-motion stimuli, using multiple frames. As an example the tolerance analysis will be applied to two widely known experiments. In this introduction we will describe the structurefrom-motion problem and the role a tolerance analysis can play in the investigation of it.A tolerance analysis of a system consists of an analysis of the degree to which the output of the system is resistant against perturbations in the input. When the output fluctuates a lot with small fluctuations in the input, it can be said that the output is not well defined. To develop a model of the visual system one can make use of an ideal detector model. By ideal detector we mean a system that reaches the optimal performance with respect to some criterion. Note that what is an ideal detector heavily depends on the definition. By comparing the performance of the visual system with an ideal detector model one can determine the extent to which the visual system approaches ideal performance.In this article we will show the use of a tolerance analysis in structure-from-motion (SfM) experiments. In SfM the input to the visual system consists of projections of the 3-D world on the retina (we discard input from other sensory systems). The fact that 3-D properties have to be inferred from projections highly limits performance in 3-D tasks. We will consider a world consisting of a number of landmarks. To obtain the 3-D structure of a number of landmarks from its projections, assumptions have to be made about the 3-D structure or the 3-D movement between the projections. Like many others, we assume that the 3-D movement is a rigid one. We assume that the projected locations of the landmarks will be available up to limited precision. More precisely, we assume that the locations available to the system are drawn from normal distributions around the actual ones. No other assumptions about the 3-D structure or the movement are being used. We will explain a method to develop an ideal detector model for rigidly moving objects given these assumptions. We will develop such a model for a situation in which the object rotates rigidly about an axis parallel to the image plane. By comparing performance of the visual system with that of the ideal detector model we can find out the degree to which it resembles the ideal detector model. Taking 2-D measures, which are more directly related to the stimulus, has several advantages over taking 3-D measures of performance ...