A neuromorphic system for video object recognition

Abstract

Automated video object recognition is a topic of emerging importance in both defense and civilian applications. This work describes an accurate and low-power neuromorphic architecture and system for real-time automated video object recognition. Our system, Neuormorphic Visual Understanding of Scenes (NEOVUS), is inspired by computational neuroscience models of feed-forward object detection and classification pipelines for processing visual data. The NEOVUS architecture is inspired by the ventral (what) and dorsal (where) streams of the mammalian visual pathway and integrates retinal processing, object detection based on form and motion modeling, and object classification based on convolutional neural networks. The object recognition performance and energy use of the NEOVUS was evaluated by the Defense Advanced Research Projects Agency (DARPA) under the Neovision2 program using three urban area video datasets collected from a mix of stationary and moving platforms. These datasets are challenging and include a large number of objects of different types in cluttered scenes, with varying illumination and occlusion conditions. In a systematic evaluation of five different teams by DARPA on these datasets, the NEOVUS demonstrated the best performance with high object recognition accuracy and the lowest energy consumption. Its energy use was three orders of magnitude lower than two independent state of the art baseline computer vision systems. The dynamic power requirement for the complete system mapped to commercial off-the-shelf (COTS) hardware that includes a 5.6 Megapixel color camera processed by object detection and classification algorithms at 30 frames per second was measured at 21.7 Watts (W), for an effective energy consumption of 5.45 nanoJoules (nJ) per bit of incoming video. These unprecedented results show that the NEOVUS has the potential to revolutionize automated video object recognition toward enabling practical low-power and mobile video processing applications.

Editorial: Hierarchical Object Representations in the Visual Cortex and Computer Vision

“…The final stage in the object recognition pathway is the inferotemporal cortex (IT), Leeds et al ( 2014 ) present an fMRI study that tries to answers the problem of how starting from simple edge-like features in V1 we obtain neurons at the top of the hierarchy that respond to complex features as parts, textures or shapes. Using feed-forward object detection and classification modeling, Khosla et al ( 2014 ) developed a neuromorphic system that also efficiently produces automated video object recognition. However, the visual system is not limited to only detecting objects, but can also detect the spatial relationships between objects and even between parts of the same object.…”

Track Everything: Limiting Prior Knowledge in Online Multi-Object Recognition

“…An image sequence data set that does contain multiple object types has been provided by the DARPA Neovision2 [27] program. This data set was collected to enable training and evaluation of Neuromorphic Vision algorithms [28], [29], [30], [31], which are a class of object recognition algorithms motivated by the emergence of bio-inspired vision sensors [32] and processing hardware (e.g. [33]).…”

The Visual Neuroscience of Robotic Grasping

“…Strongly biologically inspired models of visual areas have been implemented, focusing mainly on the primary visual cortex (see e.g. Bio-inspired systems for object recognition (Khosla et al 2014) and biological motion detection (Yousefi and Loo 2014a, b) are also available Complex agent-object interactionssuch as in grasping actions-are not a usual target of computational models. Bio-inspired systems for object recognition (Khosla et al 2014) and biological motion detection (Yousefi and Loo 2014a, b) are also available Complex agent-object interactionssuch as in grasping actions-are not a usual target of computational models.…”