Invariance of visual operations at the level of receptive fields

Abstract

The brain is able to maintain a stable perception although the visual stimuli vary substantially on the retina due to geometric transformations and lighting variations in the environment. This paper presents a theory for achieving basic invariance properties already at the level of receptive fields. Specifically, the presented framework comprises (i) local scaling transformations caused by objects of different size and at different distances to the observer, (ii) locally linearized image deformations caused by variations in the viewing direction in relation to the object, (iii) locally linearized relative motions between the object and the observer and (iv) local multiplicative intensity transformations caused by illumination variations. The receptive field model can be derived by necessity from symmetry properties of the environment and leads to predictions about receptive field profiles in good agreement with receptive field profiles measured by cell recordings in mammalian vision. Indeed, the receptive field profiles in the retina, LGN and V1 are close to ideal to what is motivated by the idealized requirements. By complementing receptive field measurements with selection mechanisms over the parameters in the receptive field families, it is shown how true invariance of receptive field responses can be obtained under scaling transformations, affine transformations and Galilean transformations. Thereby, the framework provides a mathematically well-founded and biologically plausible model for how basic invariance properties can be achieved already at the level of receptive fields and support invariant recognition of objects and events under variations in viewpoint, retinal size, object motion and illumination. The theory can explain the different shapes of receptive field profiles found in biological vision, which are tuned to different sizes and orientations in the image domain as well as to different image velocities in space-time, from a requirement that the visual system should be invariant to the natural types of image transformations that occur in its environment.

Spatio-Temporal Scale Selection in Video Data

“…Then, the image operations can be made truly covariant under local, regional or global Galilean image transformations [67,71] and allow for a more explicit separation of spatio-temporal receptive field responses that correspond to more complex spatio-temporal image structures than local Galilean motions.…”

“…Cell recordings from neurones in the primary visual cortex have shown that there are spatio-temporal receptive fields tuned to different sizes and orientations in the image domain, to different integration times over the temporal domain as well as to different image velocities in space-time [12,13,32,33]. Interestingly, the shapes of the spatio-temporal receptive field families that have been measured in biological vision can furthermore be explained by normative theories of visual receptive fields [69,71,75,78], whose axiomatic derivation is based on structural properties of the environment in combination with assumptions about the internal structure of an idealized vision system to ensure a consistent treatment of image representations over multiple spatio-temporal scales.…”

Image Matching Using Generalized Scale-Space Interest Points

“…Thereby, the detection scale can be used for defining a local scale normalized reference frame around the interest point [109,111] implying that image descriptors defined relative to such a scale-normalized reference frame will also be provably scale invariant.…”

“…A general theoretical framework for how local receptive field responses, as used in the SIFT and SURF descriptors and their extensions or analogues to colour images and spatiotemporal image data, can constitute the basis for computing inherent properties of objects to support invariant recognition under natural image transformations is presented in (Lindeberg [106,109]) including relations to receptive fields in biological vision.…”

Covering the Space of Tilts. Application to Affine Invariant Image Comparison

“…The idea of affine shape adaptation underlying one of the methodologies for achieving affine invariance, was then in turn used as a base for the work on affine invariant interest points and affine invariant matching in [28,6,37,38,58,57,56]. The notion of an affine invariant reference frame was further developed in [30,31]. Nevertheless, to the best of our knowledge, the direct constructions of affine invariant descriptors as fixed points for an iterative affine normalization process have never found a mathematical justification.…”