Event-Based Processing of Single Photon Avalanche Diode Sensors

Afshar, Saeed; Hamilton, Tara Julia; Davis, Langdon; Schaik, André van; Delic, Dennis

doi:10.1109/jsen.2020.2979761

Cited by 16 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although it is possible to create intensity images from event data in post-processing [3], our method natively captures scene intensities at single-photon resolution: the "events" in our sensing modality are individual photons. The notion of using photon detections as "spiking events" has also been explored in the context of biologically inspired vision sensors [30,1]. We derive flux changepoints from the high-resolution photon timestamp data.…”

Section: Event-based Vision Sensorsmentioning

confidence: 99%

Motion Adaptive Deblurring with Single-Photon Cameras

Seets¹,

Ingle²,

Laurenzis³

et al. 2020

Preprint

View full text Add to dashboard Cite

Single-photon avalanche diodes (SPADs) are a rapidly developing image sensing technology with extreme low-light sensitivity and picosecond timing resolution. These unique capabilities have enabled SPADs to be used in applications like LiDAR, non-line-of-sight imaging and fluorescence microscopy that require imaging in photon-starved scenarios. In this work we harness these capabilities for dealing with motion blur in a passive imaging setting in low illumination conditions. Our key insight is that the data captured by a SPAD array camera can be represented as a 3D spatio-temporal tensor of photon detection events which can be integrated along arbitrary spatio-temporal trajectories with dynamically varying integration windows, depending on scene motion. We propose an algorithm that estimates pixel motion from photon timestamp data and dynamically adapts the integration windows to minimize motion blur. Our simulation results show the applicability of this algorithm to a variety of motion profiles including translation, rotation and local object motion. We also demonstrate the real-world feasibility of our method on data captured using a 32 × 32 SPAD camera.

show abstract

Section: Event-based Vision Sensorsmentioning

confidence: 99%

Motion Adaptive Deblurring with Single-Photon Cameras

Seets¹,

Ingle²,

Laurenzis³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Single photon avalanche diode (SPAD) imager can capture high-speed 3D images at very low light levels due to its single photon counting capability. With excellent timing response and the ability to produce a digital pulse from a single detected photon, SPADs find a wide range of applications including robotics, bioimaging, facial recognition, and automotive light-detection-and-ranging (LiDAR) for autonomous vehicles [1]- [8]. Traditional SPAD imagers encode the time of flight information of incoming photons using a time to digital converter (TDC) in direct time of flight form (Fig.…”

Section: Introductionmentioning

confidence: 99%

“…Therefore, existing vision sensors and SPAD imagers still rely on off-chip processing of digital address events generated from AER readout, resulting in considerable computational overhead and latency. A frame-based SPAD sensor array with event-based processing in FPGA was reported in [8], but it requires additional steps to convert frames into events, adding complexity to the system. Additionally, all on-chip processing implementations and event based processing in [8] are based on FPGA-based boards that are power and area-intensive.…”

Section: Introductionmentioning

confidence: 99%

“…A frame-based SPAD sensor array with event-based processing in FPGA was reported in [8], but it requires additional steps to convert frames into events, adding complexity to the system. Additionally, all on-chip processing implementations and event based processing in [8] are based on FPGA-based boards that are power and area-intensive. Hence, there exists a need for an efficient on-chip processing methodology for the data generated by AER readout of the SPAD based vision sensor system.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Single Chip SPAD Based Vision Sensing System With Integrated Memristive Spiking Neuromorphic Processing

et al. 2023

View full text Add to dashboard Cite

This paper presents a new scalable 8 × 8 single photon avalanche diode (SPAD) based vision sensor with integrated spiking neuromorphic system on a single chip. The proposed vision sensing system adopts the benefits of SPAD's high quantum efficiency and energy efficiency of memristive spiking neuromorphic processing. The SPAD based vision sensor includes biologically inspired address event representation (AER) readout to generate asynchronous digital address events at the output reducing computation and making it suitable to process directly with integrated on-chip spiking neuromorphic system in a faster and more energy efficient way. A novel on-chip interface is designed to convert the output events of a SPAD-based event sensor into temporally coded spikes (TCS) that enable on-chip processing with integrated spiking neuromorphic system. We have tested the prototype vision sensing system for imaging characters by SPAD based vision sensor and recognizing them using the integrated memristive spiking neuromorphic system. To help with the evaluation, we have built a complete temporal pulses data set from simulating the SPAD vision sensor with AER readout in imaging characters and applied directly to integrated spiking neuromorphic system via designed novel on-chip interface. The achieved accuracy is 89.54% with a power consumption of 316 𝜇W for the memristive neuromorphic processor. The SPAD based vision sensor exhibits array-level dynamic range of 148 dB with a power consumption of 2.8 mW. The designed SPAD-based vision sensing system with an integrated spiking neuromorphic system on a single chip shows great promise for robotics, autonomous vehicles, health, and security applications.

show abstract

“…These devices enable high-speed 3D imaging, especially when featuring on-chip processing such as histogramming [15]. Moreover, dToF sensors can maintain a high accuracy over hundreds of meters even under sunlight and do not suffer from multipath issues like iToF, making them the basis of LiDAR devices [16,17]. Indeed, automotive LiDAR and autonomous driving have motivated a lot of research into both dToF sensors, and object detection based on depth data as considered here.…”

Section: Introductionmentioning

confidence: 99%

High-speed object detection with a single-photon time-of-flight image sensor

Mora-Martín,

Turpin,

Ruget

et al. 2021

Preprint

View full text Add to dashboard Cite

3D time-of-flight (ToF) imaging is used in a variety of applications such as augmented reality (AR), computer interfaces, robotics and autonomous systems. Single-photon avalanche diodes (SPADs) are one of the enabling technologies providing accurate depth data even over long ranges. By developing SPADs in array format with integrated processing combined with pulsed, flood-type illumination, high-speed 3D capture is possible. However, array sizes tend to be relatively small, limiting the lateral resolution of the resulting depth maps, and, consequently, the information that can be extracted from the image for applications such as object detection. In this paper, we demonstrate that these limitations can be overcome through the use of convolutional neural networks (CNNs) for high-performance object detection. We present outdoor results from a portable SPAD camera system that outputs 16-bin photon timing histograms with 64×32 spatial resolution. The results, obtained with exposure times down to 2 ms (equivalent to 500 FPS) and in signal-to-background (SBR) ratios as low as 0.05, point to the advantages of providing the CNN with full histogram data rather than point clouds alone. Alternatively, a combination of point cloud and active intensity data may be used as input, for a similar level of performance. In either case, the GPU-accelerated processing time is less than 1 ms per frame, leading to an overall latency (image acquisition plus processing) in the millisecond range, making the results relevant for safety-critical computer vision applications which would benefit from faster than human reaction times.

show abstract

Event-Based Processing of Single Photon Avalanche Diode Sensors

Cited by 16 publications

References 20 publications

Motion Adaptive Deblurring with Single-Photon Cameras

Motion Adaptive Deblurring with Single-Photon Cameras

A Single Chip SPAD Based Vision Sensing System With Integrated Memristive Spiking Neuromorphic Processing

High-speed object detection with a single-photon time-of-flight image sensor

Contact Info

Product

Resources

About