HARP: Hierarchical Attention Oriented Region-Based Processing for High-Performance Computation in Vision Sensor

Abstract

Cameras are widely adopted for high image quality with the rapid advancement of complementary metal-oxide-semiconductor (CMOS) image sensors while offloading vision applications’ computation to the cloud. It raises concern for time-critical applications such as autonomous driving, surveillance, and defense systems since moving pixels from the sensor’s focal plane are expensive. This paper presents a hardware architecture for smart cameras that understands the salient regions from an image frame and then performs high-level inference computation for sensor-level information creation instead of transporting raw pixels. A visual attention-oriented computational strategy helps to filter a significant amount of redundant spatiotemporal data collected at the focal plane. A computationally expensive learning model is then applied to the interesting regions of the image. The hierarchical processing in the pixels’ data path demonstrates a bottom-up architecture with massive parallelism and gives high throughput by exploiting the large bandwidth available at the image source. We prototype the model in field-programmable gate array (FPGA) and application-specific integrated circuit (ASIC) for integrating with a pixel-parallel image sensor. The experiment results show that our approach achieves significant speedup while in certain conditions exhibits up to 45% more energy efficiency with the attention-oriented processing. Although there is an area overhead for inheriting attention-oriented processing, the achieved performance based on energy consumption, latency, and memory utilization overcomes that limitation.

Integrating Image Quality Enhancement Methods and Deep Learning Techniques for Remote Sensing Scene Classification

“…In future work, we will apply region-based processing [4] to eliminate background information and obtain faster convergence for model training and inference.…”

“…Consequently, it is difficult to achieve good results with the scene classification task. Using the Sobel edge detection algorithm and Gaussian smoothing function to identify the relevant region in an image, Bhowmik et al [4] suggested region-based processing to discard background information and obtain a faster converge for deep learning model training. The image quality assessment method is used to quantify the level of image accuracy, which can be classified as subjective and objective methods.…”

Data-Driven Channel Pruning towards Local Binary Convolution Inverse Bottleneck Network Based on Squeeze-and-Excitation Optimization Weights

“…In CNN, based on the original attention mechanism applied to the human visual system [31,32], the spatial attention mechanism was applied to it very early. For example, in the original picture information, focus on the area that needs more attention [33][34][35], and find the relationship weight of any pixel in the image to the current pixel from the global information [36]. In addition to spatial attention, many studies have focused on the attention mechanism of the convolution channel [37] in CNN.…”

“…In recent studies, the attention mechanism has been widely used in Convolutional Neural Network (CNN). From the spatial attention mechanism [31][32][33][34][35][36], channel attention mechanism [37], and part of the research used a mixed attention mechanism [38][39][40]. The attention mechanism itself calculates the features extracted by the model and adds more weight to the more important features of the result.…”

FPGA-Based Pedestrian Detection for Collision Prediction System

“…Image sensors (cameras) are an attractive choice because they provide a large amount of environmental information in pixels. This technology is constantly evolving [8,9] and, today, it is possible to have high-resolution cameras with high sampling rates [10][11][12]. Higher resolution cameras facilitate the detection task and allow the system to detect distant pedestrians, as they will be encoded in a greater number of pixels.…”