EBBINNOT: A Hardware-Efficient Hybrid Event-Frame Tracker for Stationary Dynamic Vision Sensors

Mohan, Vivek; Singla, Deepak; Pulluri, Tarun; Ussa, Andres; Gopalakrishnan, P. K.; Sun, Pao-Sheng; Ramesh, Bharath; Basu, Arindam

doi:10.1109/jiot.2022.3178120

Cited by 12 publications

(10 citation statements)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The dynamic gain in luminosity (Figure 3a) of the tandem LED device can be utilized for noise filtering and contrast enhancement. Removing noise is a critical pre‐processing step for image processing to improve accuracy in region proposal [ 20 ] and object recognition. Conventionally, in the von Neumann architecture, the image pixels are read out and filtered digitally (e.g., by median filtering), where a bottleneck can lead to poor real‐time capabilities.…”

Section: Resultsmentioning

confidence: 99%

Ion‐Mediated Recombination Dynamics in Perovskite‐Based Memory Light‐Emitting Diodes for Neuromorphic Control Systems

Yantara,

Ng,

Sharma

et al. 2023

Advanced Materials

Self Cite

View full text Add to dashboard Cite

Neuromorphic devices can help perform memory‐heavy tasks more efficiently due to the co‐localization of memory and computing. In biological systems, fast dynamics are necessary for rapid communication, while slow dynamics aid in the amplification of signals over noise and regulatory processes such as adaptation‐ such dual dynamics are key for neuromorphic control systems. Halide perovskites exhibit much more complex phenomena than conventional semiconductors due to their coupled ionic, electronic, and optical properties which result in modulatable drift, diffusion of ions, carriers, and radiative recombination dynamics. We exploit this to engineer a dual‐emitter tandem device with the requisite dual slow‐fast dynamics. Here, we introduce a perovskite‐organic tandem light‐emitting diode capable of modulating its emission spectrum and intensity owing to the ion‐mediated recombination zone modulation between the green‐emitting quasi‐2D perovskite layer and red‐emitting organic layer. Frequency‐dependent response and high dynamic range memory of emission intensity and spectra in a light‐emitting diode are demonstrated. Utilizing the emissive read‐out, image contrast enhancement as a neuromorphic pre‐processing step to improve pattern recognition capabilities is illustrated. As proof of concept using the device's slow‐fast dynamics, an inhibition of return mechanism is physically emulated.This article is protected by copyright. All rights reserved

show abstract

Section: Resultsmentioning

confidence: 99%

Ion‐Mediated Recombination Dynamics in Perovskite‐Based Memory Light‐Emitting Diodes for Neuromorphic Control Systems

Yantara,

Ng,

Sharma

et al. 2023

Advanced Materials

Self Cite

View full text Add to dashboard Cite

show abstract

“…Low-cost reconstruction approaches have been demonstrated in Liu and Delbruck (2022) for optical flow calculation where the binary intensity frame is generated by event accumulation followed by block matching for calculating the local optical flow. Mohan et al (2022) uses event accumulated binary frames for traffic monitoring for detecting moving cars by a stationary event camera. However, our work requires a frame-based accurate target detection using CNN for maintaining the overall accuracy of the system.…”

Section: Prey Detection Via Cnnsmentioning

confidence: 99%

“…Event camera-based moving object tracking problem has also been addressed using model-based approaches like cluster detection (Delbruck and Lang, 2013), corner detection (Vasco et al, 2016), ICP (Ni et al, 2012), region of interest tracking (Mohan et al, 2022), etc. However, these works operate with either a stationary camera or stationary environment as opposed to independently moving prey and predator in this case.…”

Section: Other Related Workmentioning

confidence: 99%

“…Histogram-based method utilized in SNN filter may get limited if directly applied to simultaneous tracking of multiple objects. Recent works have demonstrated region proposal on low-cost event-accumulated binary images followed by multi-object tracking even in presence of occlusion showing low computation and memory costs (Acharya et al, 2019;Mohan et al, 2022). Customized circuits for this application (Bose and Basu, 2022) demonstrate high throughput and energy efficiency.…”

Section: Potential Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

Bio-mimetic high-speed target localization with fused frame and event vision for edge application

et al. 2022

View full text Add to dashboard Cite

Evolution has honed predatory skills in the natural world where localizing and intercepting fast-moving prey is required. The current generation of robotic systems mimics these biological systems using deep learning. High-speed processing of the camera frames using convolutional neural networks (CNN) (frame pipeline) on such constrained aerial edge-robots gets resource-limited. Adding more compute resources also eventually limits the throughput at the frame rate of the camera as frame-only traditional systems fail to capture the detailed temporal dynamics of the environment. Bio-inspired event cameras and spiking neural networks (SNN) provide an asynchronous sensor-processor pair (event pipeline) capturing the continuous temporal details of the scene for high-speed but lag in terms of accuracy. In this work, we propose a target localization system combining event-camera and SNN-based high-speed target estimation and frame-based camera and CNN-driven reliable object detection by fusing complementary spatio-temporal prowess of event and frame pipelines. One of our main contributions involves the design of an SNN filter that borrows from the neural mechanism for ego-motion cancelation in houseflies. It fuses the vestibular sensors with the vision to cancel the activity corresponding to the predator's self-motion. We also integrate the neuro-inspired multi-pipeline processing with task-optimized multi-neuronal pathway structure in primates and insects. The system is validated to outperform CNN-only processing using prey-predator drone simulations in realistic 3D virtual environments. The system is then demonstrated in a real-world multi-drone set-up with emulated event data. Subsequently, we use recorded actual sensory data from multi-camera and inertial measurement unit (IMU) assembly to show desired working while tolerating the realistic noise in vision and IMU sensors. We analyze the design space to identify optimal parameters for spiking neurons, CNN models, and for checking their effect on the performance metrics of the fused system. Finally, we map the throughput controlling SNN and fusion network on edge-compatible Zynq-7000 FPGA to show a potential 264 outputs per second even at constrained resource availability. This work may open new research directions by coupling multiple sensing and processing modalities inspired by discoveries in neuroscience to break fundamental trade-offs in frame-based computer vision1.

show abstract

“…Event-based binary image (EBBI) is created by a hybrid approach [39] during the time interval t F = 66ms. The raw image captured by the NVS inherently contains lots of noise pixels characterized by abnormal firing rates because of a variety of factors such as thermal noise in transistors, shot noise and junction leakage current in some of the photodiodes [40][41][42][43].…”

Section: Proposed Systemmentioning

confidence: 99%

Low-power circuits for neuromorphic vision sensor based internet of video things

Zhang¹

View full text Add to dashboard Cite

I would not have been able to finish my Ph.D. study and have led an enriched research life. He is always considerate and kind. Due to his trust in me, I have been allowed to explore various research topics. I am amazed and inspired by his immense inter-disciplinary knowledge as well as the endless passion for research. He not only improved my problem-solving skills but also taught me to communicate with others effectively. I am incredibly fortunate to have him as a mentor in both academic and non-academic life. He is really a great role model, and I am forever thankful to him for the opportunity he provided in the past years. Special thanks to my supervisor Prof. Gwee Bah Hwee for the final support of my thesis submission and all his kind help for the last journey of my Ph.D. period.

show abstract

EBBINNOT: A Hardware-Efficient Hybrid Event-Frame Tracker for Stationary Dynamic Vision Sensors

Cited by 12 publications

References 68 publications

Ion‐Mediated Recombination Dynamics in Perovskite‐Based Memory Light‐Emitting Diodes for Neuromorphic Control Systems

Ion‐Mediated Recombination Dynamics in Perovskite‐Based Memory Light‐Emitting Diodes for Neuromorphic Control Systems

Bio-mimetic high-speed target localization with fused frame and event vision for edge application

Low-power circuits for neuromorphic vision sensor based internet of video things

Contact Info

Product

Resources

About