A 58.6mW real-time programmable object detector with multi-scale multi-object support using deformable parts model on 1920×1080 video at 30fps

Abstract

This paper presents a programmable, energy-efficient and realtime object detection accelerator using deformable parts models (DPM), with 2x higher accuracy than traditional rigid body models. With 8 deformable parts detection, three methods are used to address the high computational complexity: classification pruning for 33x fewer parts classification, vector quantization for 15x memory size reduction, and feature basis projection for 2x reduction of the cost of each classification. The chip is implemented in 65nm CMOS technology, and can process HD (1920x1080) images at 30fps without any off-chip storage while consuming only 58.6mW (0.94nJ/pixel, 1168 GOPS/W). The chip has two classification engines to simultaneously detect two different classes of objects. With a tested high throughput of 60fps, the classification engines can be time multiplexed to detect even more than two object classes. It is energy scalable by changing the pruning factor or disabling the parts classification. Keywords: DPM, object detection, basis projection, pruning. Introduction Object detection is critical to many embedded applications that require low power and real-time processing. For example, low latency and HD images are important for autonomous control to react quickly to fast approaching objects, while low energy consumption is essential due to battery and heat limitations. Object detection involves not only classification/recognition, but also localization, which is achieved by sliding a window of a pretrained model over an image. For multi-scale detection, the window slides over an image pyramid (multiple downscaled copies of the image). Multi-scale detection is very challenging as the image pyramid results in a data expansion, which can be more than a 100x in HD images. The high computational complexity of object detection processing necessitates fast hardware implementations [1] to enable real-time processing.This paper presents a complete object detection accelerator using DPM [2] with a root and 8 parts model as shown in Fig. 1. DPM results in double the detection accuracy compared to rigid template (root only) detection. The 8 parts account for deformation such that a single model can detect objects at different poses ( Fig. 6) and increase detection confidence. However, this accuracy comes with a classification overhead of 35x more multiplications (i.e. DPM classification consumes 80% of a single detector power), making multi-object detection a challenge. A software-based DPM object detector is described in [3], which enables detection for 500x500 images at 30fps but requires a powerful fully loaded Xeon 6-core processor and 32GB of memory. In this work, the classification overhead is significantly reduced by two main techniques: Classification pruning with vector quantization (VQ) for selective part processing.  Feature basis projection for sparse multiplications.Architecture Overview Fig. 2 shows the block diagram of our detector architecture, including histogram of oriented gradients (HOG) feature pyramid gen...

Towards closing the energy gap between HOG and CNN features for embedded vision

“…For example, the CONV layer weights from AlexNet can either be hard-wired in the multipliers, or stored in on-chip SRAM or ROM. This is not feasible if the available hardware resources are constrained to the level of the HOG design [7], i.e., 1000 kgates with 150 kB SRAM. Assuming each input and weight value take 1 byte, only 10k multipliers with fixed…”

“…In this paper, we will provide an in-depth analysis on the causes for the energy gap between hand-crafted and learned features. We use results from two actual chip designs: [7] implements the hand-crafted feature using HOG, and [8] implements the learned feature using CNN. Both chips use 65nm CMOS technology and have similar hardware resource utilization in terms of logic gate count and memory capacity.…”

A 58.6 mW 30 Frames/s Real-Time Programmable Multiobject Detection Accelerator With Deformable Parts Models on Full HD $1920\times 1080$ Videos

“…However, this comes at the cost of 35× more computation compared to rigid object detection [10]. This overhead comes from four main factors: 3× larger model size with the parts filters, 4× larger image pyramid size to support parts classification at twice the image resolution relative to the root classification, 1.5× increase due to the deformation computation, and finally 2× increase due to the fact that two DPM models are used (original and flipped version).…”

“…For multi-scale detection, the window scans an image pyramid (multiple downscaled versions of the image). Multi-scale detection increases the required computation as the image pyramid leads to data expansion, which can be a 100× increase in the number of pixels for a full HD image [10]. In classification, a pre-trained model that captures the characteristics of the target object is used at each sliding window position to label it as a true or a false object.…”

“…The deformable parts models (DPM) algorithm [20], which is based on HOG features, has demonstrated high detection accuracy. DPM doubles the detection accuracy compared to rigid object detection because of the relatively larger and more flexible models [10]. Lately, convolutional neural networks (CNN) have been widely used in large scale classification systems.…”