Canonical circuit computations for computer vision

Abstract

Advanced computer vision mechanisms have been inspired by neuroscientific findings. However, with the focus on improving benchmark achievements, technical solutions have been shaped by application and engineering constraints. This includes the training of neural networks which led to the development of feature detectors optimally suited to the application domain. However, the limitations of such approaches motivate the need to identify computational principles, or motifs, in biological vision that can enable further foundational advances in machine vision. We propose to utilize structural and functional principles of neural systems that have been largely overlooked. They potentially provide new inspirations for computer vision mechanisms and models. Recurrent feedforward, lateral, and feedback interactions characterize general principles underlying processing in mammals. We derive a formal specification of core computational motifs that utilize these principles. These are combined to define model mechanisms for visual shape and motion processing. We demonstrate how such a framework can be adopted to run on neuromorphic brain-inspired hardware platforms and can be extended to automatically adapt to environment statistics. We argue that the identified principles and their formalization inspires sophisticated computational mechanisms with improved explanatory scope. These and other elaborated, biologically inspired models can be employed to design computer vision solutions for different tasks and they can be used to advance neural network architectures of learning.

Teaching deep networks to see shape: Lessons from a simplified visual world

“…This “informative dropout” reduced the reliance of networks on texture and increased shape bias. These results are promising and lend support to recent calls for more biologically inspired components in deep networks [12, 13]. Competition is an especially promising candidate for incorporation into deep networks for several reasons: its cortical equivalent - normalization - is well documented and pervasive [42] and there are theories about the computational role it could play in vision by sparsifying and whitening representations [43, 44].…”

“…Competition is an especially promising candidate for incorporation into deep networks for several reasons: its cortical equivalentnormalization -is well documented and pervasive [42] and there are theories about the computational role it could play in vision by sparsifying and whitening representations [43,44]. In addition, it may facilitate the inclusion of other biologically inspired components, such as long-range connections, by stabilizing the system [13,45]. However, spatial competition alone is probably not enough to enable deep networks to process shape.…”

“…Another common feature in models of biological shape processing is spatial competition [13, 57, 58]. For this reason, we also included a convolutional network with a simple spatial competition mechanism ( spcConvNet ) in our analysis.…”

“…Even though a deep network may assign similar labels to images as a human participant in an experiment, it may use entirely different mechanisms and strategies to do so. While deep networks share some similarities with visual cortex, the latter makes use of many structural design principles that have not yet been adopted in deep networks, such as feedback connections or lateral grouping [13]. Indeed, several lines of evidence indicate that deep networks process images very differently from humans: they are susceptible to adversarial attacks [14], less robust to image corruptions [6, 15], and do not show evidence of perceptual grouping phenomena that play key roles in human perception [12].…”

Cellular mechanisms of cooperative context-sensitive predictive inference

“…The two-point neuron view is beginning to inspire major advances in computer vision ( Schmid et al, 2023 ) and in neuromorphic machine learning algorithms that are far more effective and energy-efficient than standard deep learning algorithms (see, e.g., Adeel et al, 2022 , 2023 ; Capone et al, 2023 ) – although whether the net effect of such advances in neuromorphic computing will be for good or ill remains to be seen. Further concrete computational studies of real-world big-data processing are now required.…”