A Camera That CNNs: Towards Embedded Neural Networks on Pixel Processor Arrays

Bose, Laurie; Dudek, Piotr; Chen, Jianing; Carey, Stephen J.; Mayol-Cuevas, Walterio

doi:10.1109/iccv.2019.00142

Cited by 35 publications

(31 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By making use of the SCAMP-5's multi-functional capability we can further improve the vehicles navigation capabilities and offer a single low power (≈ 2W) sensor that can provide a wide range of functionality. This has been demonstrated by works on HDR imaging (Martel et al, 2016 ), feature extraction (Chen et al, 2017 ), neural network classifiers (Bose et al, 2019 ), and other on-sensor algorithms which can potentially be used to enhance future VO functionality, creating extremely capable, low weight, low power multi functional sensors. As a proof of concept, this paper demonstrates the potential for use of PPAs in GNSS denied or challenging environments.…”

Section: Discussionmentioning

confidence: 99%

Visual Odometry Using Pixel Processor Arrays for Unmanned Aerial Systems in GPS Denied Environments

et al. 2020

Self Cite

View full text Add to dashboard Cite

Environments in which Global Positioning Systems (GPS), or more generally Global Navigation Satellite System (GNSS), signals are denied or degraded pose problems for the guidance, navigation, and control of autonomous systems. This can make operating in hostile GNSS-Impaired environments, such as indoors, or in urban and natural canyons, impossible or extremely difficult. Pixel Processor Array (PPA) cameras-in conjunction with other on-board sensors-can be used to address this problem, aiding in tracking, localization, and control. In this paper we demonstrate the use of a PPA device-the SCAMP vision chip-combining perception and compute capabilities on the same device for aiding in real-time navigation and control of aerial robots. A PPA consists of an array of Processing Elements (PEs), each of which features light capture, processing, and storage capabilities. This allows various image processing tasks to be efficiently performed directly on the sensor itself. Within this paper we demonstrate visual odometry and target identification running concurrently on-board a single PPA vision chip at a combined frequency in the region of 400 Hz. Results from outdoor multirotor test flights are given along with comparisons against baseline GPS results. The SCAMP PPA's High Dynamic Range (HDR) and ability to run multiple algorithms at adaptive rates makes the sensor well suited for addressing outdoor flight of small UAS in GNSS challenging or denied environments. HDR allows operation to continue during the transition from indoor to outdoor environments, and in other situations where there are significant variations in light levels. Additionally, the PPA only needs to output specific information such as the optic flow and target position, rather than having to output entire images. This significantly reduces the bandwidth required for communication between the sensor and on-board flight computer, enabling high frame rate, low power operation.

show abstract

Section: Discussionmentioning

confidence: 99%

Visual Odometry Using Pixel Processor Arrays for Unmanned Aerial Systems in GPS Denied Environments

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Hence, it is essential to overcome the limitations in data transportation time, and there are some works for better utilize the bandwidth available at the image source [ 5 , 25 , 26 ]. A programmable vision chip Scamp5d is proposed by Chen et al which has SIMD-like parallel processing directly at the focal plane [ 25 ].…”

Section: Related Workmentioning

confidence: 99%

“…These signal processors compute comparatively simple operations, such as extract temporal contrast in each pixel, low-level window-based imaging processing applications [ 4 ]. Here, the added benefits are massive bandwidth available at the sensor interface, and it enables us to meet the size, weight, and power (SWaP) constraints [ 5 ].…”

Section: Introductionmentioning

confidence: 99%

HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor

Bhowmik

Pantho

Bobda

2021

Sensors

View full text Add to dashboard Cite

Cameras are widely adopted for high image quality with the rapid advancement of complementary metal-oxide-semiconductor (CMOS) image sensors while offloading vision applications’ computation to the cloud. It raises concern for time-critical applications such as autonomous driving, surveillance, and defense systems since moving pixels from the sensor’s focal plane are expensive. This paper presents a hardware architecture for smart cameras that understands the salient regions from an image frame and then performs high-level inference computation for sensor-level information creation instead of transporting raw pixels. A visual attention-oriented computational strategy helps to filter a significant amount of redundant spatiotemporal data collected at the focal plane. A computationally expensive learning model is then applied to the interesting regions of the image. The hierarchical processing in the pixels’ data path demonstrates a bottom-up architecture with massive parallelism and gives high throughput by exploiting the large bandwidth available at the image source. We prototype the model in field-programmable gate array (FPGA) and application-specific integrated circuit (ASIC) for integrating with a pixel-parallel image sensor. The experiment results show that our approach achieves significant speedup while in certain conditions exhibits up to 45% more energy efficiency with the attention-oriented processing. Although there is an area overhead for inheriting attention-oriented processing, the achieved performance based on energy consumption, latency, and memory utilization overcomes that limitation.

show abstract

“…All neurons belonging to the same input/output channel pair have the same weight. Hardware implementations of CNNs making use of this constraint, holding each weight in their memory only once and applying them to different sections of the input either by broadcasting (Bose et al, 2019) or sequentially, as, for example, on GPUs (Chetlur et al, 2014). On GPUs, the convolution operation is commonly recast as a highly optimized general matrix multiple between the filters and a copied and tiled input image (though many variants of GPU convolutions exist).…”

Section: Hardware Constraintmentioning

confidence: 99%

“…An example of a hardware implementation of spatially localized processing is the SCAMP-5 sensor/processor array (Carey et al, 2013). The nearest-neighbor communication structure of this chip allows for an efficient pixelparallel implementation of convolution filters if the filters are small (Bose et al, 2019). In GPU implementations of CNNs, small filters need fewer replications of each source pixel (as well as less memory for the filters themselves).…”

Section: Hardware Constraintmentioning

confidence: 99%

Neuromorphic Systems Design by Matching Inductive Biases to Hardware Constraints

2020

View full text Add to dashboard Cite

Neuromorphic systems are designed with careful consideration of the physical properties of the computational substrate they use. Neuromorphic engineers often exploit physical phenomena to directly implement a desired functionality, enabled by "the isomorphism between physical processes in different media" (Douglas et al., 1995). This bottom-up design methodology could be described as matching computational primitives to physical phenomena. In this paper, we propose a top-down counterpart to the bottom-up approach to neuromorphic design. Our top-down approach, termed "bias matching," is to match the inductive biases required in a learning system to the hardware constraints of its implementation; a well-known example is enforcing translation equivariance in a neural network by tying weights (replacing vector-matrix multiplications with convolutions), which reduces memory requirements. We give numerous examples from the literature and explain how they can be understood from this perspective. Furthermore, we propose novel network designs based on this approach in the context of collaborative filtering. Our simulation results underline our central conclusions: additional hardware constraints can improve the predictions of a Machine Learning system, and understanding the inductive biases that underlie these performance gains can be useful in finding applications for a given constraint.

show abstract

A Camera That CNNs: Towards Embedded Neural Networks on Pixel Processor Arrays

Cited by 35 publications

References 26 publications

Visual Odometry Using Pixel Processor Arrays for Unmanned Aerial Systems in GPS Denied Environments

Visual Odometry Using Pixel Processor Arrays for Unmanned Aerial Systems in GPS Denied Environments

HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor

Neuromorphic Systems Design by Matching Inductive Biases to Hardware Constraints

Contact Info

Product

Resources

About