A Camera with Brain – Embedding Machine Learning in 3D Sensors

Mudassar, Burhan Ahmad; Saha, Priyabrata; Long, Yun; Amir, Muhammad; Gebhardt, Evan; Na, Taesik; Ko, Jong Hwan; Wolf, Marilyn; Mukhopadhyay, Saibal

doi:10.23919/date.2019.8715258

Cited by 6 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…putation and data transmission [30,37,38,43,54,55,68,74,92]. Classic work such as fast R-CNN [37] use dedicated region proposal networks that are computationally heavy.…”

Section: Related Workmentioning

confidence: 99%

“…Classic work such as fast R-CNN [37] use dedicated region proposal networks that are computationally heavy. Other approaches use simple extrapolation [30,68,92], which we find insufficient for eye tracking, because the objects (eyes) move rapidly. Many image sensors provide an ROI output mode [4,5], but rely on users to provide the ROI coordinates.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Real-Time Gaze Tracking with Event-Driven Eye Segmentation

Ye¹,

Goulding-Hotta²,

Khan³

et al. 2022

Preprint

View full text Add to dashboard Cite

Workflow of our gaze tracking algorithm. The algorithm provides an Auto ROI mode (akin to the conventional "3A" algorithms in a camera) by emulating an event camera in software and using the events to predict the Region of Interest (ROI). Performing Auto ROI in emerging compute-capable image sensors further reduces the data transmission energy. Right: Our algorithm achieves sub-0.5°gaze accuracies (both horizontal and vertical directions) using only 30K parameters, an order of magnitude smaller than other algorithms. Marker sizes in the figure are proportional to the number of parameters. The algorithm operates at over 30 Hz on a mobile Jetson Xavier processor.

show abstract

“…putation and data transmission [30,37,38,43,54,55,68,74,92]. Classic work such as fast R-CNN [37] use dedicated region proposal networks that are computationally heavy.…”

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Real-Time Gaze Tracking with Event-Driven Eye Segmentation

Ye¹,

Goulding-Hotta²,

Khan³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Multilayer integration, with the sensor stacked with ISP and memory layers, allows radical reductions in size and power [106,107]. Recent studies even integrate neural processors into such systems [108,109]. In the longer term one can expect sensors that directly integrate compression and neural sampling in the focal plane architecture, closing a loop back to artificial retinas developed in an earlier cycle of neural computing [110].…”

Section: Physical Structure Of Smart Camerasmentioning

confidence: 99%

Smart Cameras

Brady,

Hu,

Wang

et al. 2020

Preprint

View full text Add to dashboard Cite

We review camera architecture in the age of artificial intelligence. Modern cameras use physical components and software to capture, compress and display image data. Over the past 5 years, deep learning solutions have become superior to traditional algorithms for each of these functions. Deep learning enables 10-100x reduction in electrical sensor power per pixel, 10x improvement in depth of field and dynamic range and 10-100x improvement in image pixel count. Deep learning enables multiframe and multiaperture solutions that fundamentally shift the goals of physical camera design. Here we review the state of the art of deep learning in camera operations and consider the impact of AI on the physical design of cameras.

show abstract