Flies and humans share a motion estimation strategy that exploits natural scene statistics

Abstract

Sighted animals extract motion information from visual scenes by processing spatiotemporal patterns of light falling on the retina. The dominant models for motion estimation exploit intensity correlations only between pairs of points in space and time. Moving natural scenes, however, contain more complex correlations. Here we show that fly and human visual systems encode the combined direction and contrast polarity of moving edges using triple correlations that enhance motion estimation in natural environments. Both species extract triple correlations with neural substrates tuned for light or dark edges, and sensitivity to specific triple correlations is retained even as light and dark edge motion signals are combined. Thus, both species separately process light and dark image contrasts to capture motion signatures that can improve estimation accuracy. This striking convergence argues that statistical structures in natural scenes have profoundly affected visual processing, driving a common computational strategy over 500 million years of evolution.

Modular Representation of Luminance Polarity in the Superficial Layers of Primary Visual Cortex

“…The idea that information from ON and OFF channels continues to be processed in parallel by cortical circuits is consistent with evidence from human psychophysical studies showing that the cortical mechanisms for encoding direction of motion and stereopsis extract these properties independently for light and dark stimuli (Mather et al, 1991; Harris and Parker, 1995; Clark et al, 2014). Furthermore, maintaining separate pathways allows the cortex to take advantage of asymmetries in ON and OFF processing, including enhanced spatial resolution and faster processing of dark stimuli (Jin et al, 2011a; Komban et al, 2011; Komban et al, 2014; Kremkow et al, 2014).…”

“…Furthermore, maintaining separate pathways allows the cortex to take advantage of asymmetries in ON and OFF processing, including enhanced spatial resolution and faster processing of dark stimuli (Jin et al, 2011a; Komban et al, 2011; Komban et al, 2014; Kremkow et al, 2014). The separate processing of light and dark stimuli appears to enhance the fidelity of motion estimation, taking advantage of correlations in natural scenes (Clark et al, 2014). Similarly, disparity tuning mechanisms are thought to benefit from distinct light and dark signals in matching the inputs from the two eyes, and in extracting environmental correlations between brightness and depth (Samonds et al, 2012).…”

Evidence and Counterevidence in Motion Perception

“…Other examples of such higherorder sensitivity have previously been reported. For instance, neurons in area MT are sensitive to the higher-order correlations created by multiple successive motion steps in the same direction (Mikami et al 1986), and both fruit flies and humans reliably extract additional motion information from scenes containing three-point diverging and converging spatiotemporal correlations that are invisible to the standard motion models and the Bours-Lankheet model (Hu and Victor 2010;Fitzgerald et al 2011;Clark et al 2014). Recent work from our laboratory suggests that networks of recurrently connected neurons are well suited to extract such higher-order statistical regularities dynamically (Richert et al 2013;Joukes et al 2014).…”

Direct Measurement of Correlation Responses in Drosophila Elementary Motion Detectors Reveals Fast Timescale Tuning

“…These correlations contain information that can be used to determine the direction and speed of the image’s motion (Fitzgerald et al, 2011; Potters and Bialek, 1994). Motion generates many kinds of correlations (Nitzany and Victor, 2014), but the strongest indicators of motion are frequently pairwise correlations in intensity over time and space (Clark et al, 2014; Fitzgerald and Clark, 2015; Fitzgerald et al, 2011; Potters and Bialek, 1994). Useful information about the direction of motion is carried by both positive and negative pairwise correlations (Clark et al, 2011; Fitzgerald and Clark, 2015).…”

“…The HRC has been so successful in part because any nonlinear combination of two inputs is likely to have a large component that can be described mathematically as a simple product (Poggio and Reichardt, 1973) (see SI). Despite this success, the purely multiplicative HRC model in Drosophila has been excluded by several lines of evidence, including the fly’s ON and OFF motion pathways (Behnia et al, 2014; Clark et al, 2011; Joesch et al, 2010; Maisak et al, 2013) and DS responses to higher-order correlations (Clark et al, 2014). Other proposed models are HRC-like, in that they amplify signals to PD motion, but differ by not invoking pure multiplication (Behnia et al, 2014; Eichner et al, 2011; Serbe et al, 2016).…”