ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks Using Angle Sensitive Pixels

Chen, Huaijin; Jayasuriya, Suren; Yang, Jiyue; Stephen, Judy; Sivaramakrishnan, Sriram; Veeraraghavan, Ashok; Molnar, Alyosha

doi:10.1109/cvpr.2016.104

Cited by 54 publications

(29 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, in [114] an Angle Sensitive Pixels sensor is used to compute the gradient of the input, which along with compression, reduces the data movement from the sensor by 10×. In addition, since the first layer of the DNN often outputs a gradient-like feature map, it maybe possible to skip the computations in the first layer, which further reduces energy consumption as discussed in [115,116].…”

Section: Sensorsmentioning

confidence: 99%

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

et al. 2017

View full text Add to dashboard Cite

Abstract-Deep neural networks (DNNs) are currently widely used for many artificial intelligence (AI) applications including computer vision, speech recognition, and robotics. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, techniques that enable efficient processing of DNNs to improve energy efficiency and throughput without sacrificing application accuracy or increasing hardware cost are critical to the wide deployment of DNNs in AI systems.This article aims to provide a comprehensive tutorial and survey about the recent advances towards the goal of enabling efficient processing of DNNs. Specifically, it will provide an overview of DNNs, discuss various hardware platforms and architectures that support DNNs, and highlight key trends in reducing the computation cost of DNNs either solely via hardware design changes or via joint hardware design and DNN algorithm changes. It will also summarize various development resources that enable researchers and practitioners to quickly get started in this field, and highlight important benchmarking metrics and design considerations that should be used for evaluating the rapidly growing number of DNN hardware designs, optionally including algorithmic co-designs, being proposed in academia and industry.The reader will take away the following concepts from this article: understand the key design considerations for DNNs; be able to evaluate different DNN hardware implementations with benchmarks and comparison metrics; understand the trade-offs between various hardware architectures and platforms; be able to evaluate the utility of various DNN design techniques for efficient processing; and understand recent implementation trends and opportunities.

show abstract

Section: Sensorsmentioning

confidence: 99%

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

et al. 2017

View full text Add to dashboard Cite

show abstract

“…It is important to note that clearly scatterers could be directly engineered onto the sensor to enhance such reconstructions in the future, which would make it similar to but much more compact than mask-based lensless imaging methods. Such techniques could obviously be extended to non-imaging computational problems including inference using deep learning and related algorithms [18][19] .…”

Section: (D) Correlation Coefficient Maps Of 343mm Calibrationmentioning

confidence: 99%

Lensless photography with only an image sensor

et al. 2017

View full text Add to dashboard Cite

Photography usually requires optics in conjunction with a recording device (an image sensor). Eliminating the optics could lead to new form factors for cameras. Here, we report a simple demonstration of imaging using a bare CMOS sensor that utilizes computation. The technique relies on the space variant point-spread functions resulting from the interaction of a point source in the field of view with the image sensor. These space-variant point-spread functions are combined with a reconstruction algorithm in order to image simple objects displayed on a discrete LED array as well as on an LCD screen. We extended the approach to video imaging at the native frame rate of the sensor. Finally, we performed experiments to analyze the parametric impact of the object distance. Improving the sensor designs and reconstruction algorithms can lead to useful cameras without optics.The optical systems of cameras in mobile devices typically constrain the overall thickness of the devices 1-2 . By eliminating the optics, it is possible to create ultra-thin cameras with interesting new form factors. Previous work in computational photography has eliminated the need for lenses by utilizing apertures in front of the image sensor 3-7 or via coherent illumination of the sample 8 . In the former case, the apertures create shadow patterns on the sensor that could be computationally recovered by solving a linear inverse problem 9 . The latter case requires coherent illumination, which is not generally applicable to imaging. In most instances, coded apertures have replaced the lenses. Microfabricated coded apertures have recently shown potential for the thinner systems, 4 with thickness on the order of millimeters. However, these apertures are absorbing and hence, exhibit relatively low transmission efficiencies. Another method utilizes holographic phase masks integrated onto the image sensor in conjunction with computation to enable simple imaging. 10,11 In this case, precise microfabrication of the mask onto the sensor is required.Another computational camera utilizes a microlens array to form a large number of partial images of the scene, which is then numerically combined to form a single image with computational refocusing [12][13][14] . Here, we report on a computational camera that is comprised of only a conventional image sensor and no other elements.Our motivation for this camera is based upon the recognition that all cameras essentially rely on the fact that the information about the object enters the aperture of the lens, the coded aperture or micro-lens array, and is recorded by the image sensor. In the case of the coded aperture and the microlens array, numerical processing is performed to represent the image for human consumption. If all optical elements are eliminated, the information from the object is still recorded by the image sensor. If appropriate reconstruction algorithms were developed, the image recorded by the sensor can be subsequently recovered for human consumption. It is analogous the multi-sensory compressive ...

show abstract

“…While this paper exploits temporal redundancy to avoid CNN layer computation, AMC also suggests opportunities for savings in the broader system. Future work can integrate camera sensors that avoid spending energy to capture redundant data [28,[62][63][64], and end-to-end visual applications can inform the system about which semantic changes are relevant for their task. A change-oriented visual system could exploit the motion vectors that hardware video codecs already produce, as recent work has done for super-resolution [26].…”

Section: Discussionmentioning

confidence: 99%

EVA²: Exploiting Temporal Redundancy in Live Computer Vision

Buckler

Bedoukian

Jayasuriya

et al. 2018

2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA)

Self Cite

View full text Add to dashboard Cite

Hardware support for deep convolutional neural networks (CNNs) is critical to advanced computer vision in mobile and embedded devices. Current designs, however, accelerate generic CNNs; they do not exploit the unique characteristics of real-time vision. We propose to use the temporal redundancy in natural video to avoid unnecessary computation on most frames. A new algorithm, activation motion compensation, detects changes in the visual input and incrementally updates a previously-computed activation. The technique takes inspiration from video compression and applies well-known motion estimation techniques to adapt to visual changes. We use an adaptive key frame rate to control the trade-off between efficiency and vision quality as the input changes. We implement the technique in hardware as an extension to state-of-the-art CNN accelerator designs. The new unit reduces the average energy per frame by 54%, 62%, and 87% for three CNNs with less than 1% loss in vision accuracy.

show abstract

ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks Using Angle Sensitive Pixels

Cited by 54 publications

References 46 publications

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Efficient Processing of Deep Neural Networks: A Tutorial and Survey

Lensless photography with only an image sensor

EVA²: Exploiting Temporal Redundancy in Live Computer Vision

Contact Info

Product

Resources

About