Adaptive Slicing Method of the Spatiotemporal Event Stream Obtained from a Dynamic Vision Sensor

Zhang, Yisa; Zhao, Yuchen; Lv, Hengyi; Feng, Yang; Liu, Hailong; Han, Chengshan

doi:10.3390/s22072614

Cited by 5 publications

(5 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To directly use the discrete characteristics of spatiotemporal data and reduce the computational complexity of the network, we use the spike neural network to directly process the spatiotemporal data. At the same time, in order to avoid target loss or motion blur in the event slice for optical flow estimation, we use the adaptive slicing method to slice the spatiotemporal data stream [ 7 ]. Then, event slices are sent into the network in discrete form, as shown in Figure 3 .…”

Section: Methodsmentioning

confidence: 99%

“… ( a ) spatiotemporal data stream of the event camera; ( b ) spiking input event representation obtained by [ 7 ]. …”

Section: Figurementioning

confidence: 99%

“…Combined with the characteristics of the event camera, The set of output events in the spatiotemporal domain is known as the spatiotemporal event stream. The spatiotemporal event stream can be defined as the following [ 7 ].

where

is an event of the spatiotemporal event stream,

denotes location of the pixel generating the event,

indicates the polarity of the change in illumination at the pixel causing the event, and 𝑡 represents the time at which the event occurred;

is the index of events in the spatiotemporal event stream, and

indicates adding the new event to the spatiotemporal data stream.…”

Section: Introductionmentioning

confidence: 99%

“…This paper also uses the learning method to predict optical flow, but it is different from the above methods. First, this paper uses the method in [ 7 ] to adaptively slice the spatiotemporal data stream and then sends the event slice into the trained spike neural network in discrete form for optical flow prediction. The slice has no information loss or motion blur, which improves the optical flow prediction accuracy.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network

Zhang

Zhao

et al. 2023

Micromachines

Self Cite

View full text Add to dashboard Cite

The advantages of an event camera, such as low power consumption, large dynamic range, and low data redundancy, enable it to shine in extreme environments where traditional image sensors are not competent, especially in high-speed moving target capture and extreme lighting conditions. Optical flow reflects the target’s movement information, and the target’s detailed movement can be obtained using the event camera’s optical flow information. However, the existing neural network methods for optical flow prediction of event cameras has the problems of extensive computation and high energy consumption in hardware implementation. The spike neural network has spatiotemporal coding characteristics, so it can be compatible with the spatiotemporal data of an event camera. Moreover, the sparse coding characteristic of the spike neural network makes it run with ultra-low power consumption on neuromorphic hardware. However, because of the algorithmic and training complexity, the spike neural network has not been applied in the prediction of the optical flow for the event camera. For this case, this paper proposes an end-to-end spike neural network to predict the optical flow of the discrete spatiotemporal data stream for the event camera. The network is trained with the spatio-temporal backpropagation method in a self-supervised way, which fully combines the spatiotemporal characteristics of the event camera while improving the network performance. Compared with the existing methods on the public dataset, the experimental results show that the method proposed in this paper is equivalent to the best existing methods in terms of optical flow prediction accuracy, and it can save 99% more power consumption than the existing algorithm, which is greatly beneficial to the hardware implementation of the event camera optical flow prediction., laying the groundwork for future low-power hardware implementation of optical flow prediction for event cameras.

show abstract

Section: Methodsmentioning

confidence: 99%

“… ( a ) spatiotemporal data stream of the event camera; ( b ) spiking input event representation obtained by [ 7 ]. …”

Section: Figurementioning

confidence: 99%

where

is an event of the spatiotemporal event stream,

denotes location of the pixel generating the event,

indicates the polarity of the change in illumination at the pixel causing the event, and 𝑡 represents the time at which the event occurred;

is the index of events in the spatiotemporal event stream, and

indicates adding the new event to the spatiotemporal data stream.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network

Zhang

Zhao

et al. 2023

Micromachines

Self Cite

View full text Add to dashboard Cite

show abstract

“…Oliveria et al (Oliveria et al, 2021) proposed an asynchronous time-based image sensor (ATIS) and an eventtriggered light intensity measurement circuit to reconstruct the pixel grey level as the change occurred. Zhang et al (Zhang et al, 2022a) created a Dynamic and Active Pixel Vision Sensor (DAVIS), a dual-mode technical route that includes an additional independent traditional image sampling circuit to compensate for DVS texture imaging flaws. It was then expanded to colour DAVIS346 (Moeys et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

Artificial intelligence-based spatio-temporal vision sensors: applications and prospects

Li,

Sun

2023

Front. Mater.

View full text Add to dashboard Cite

Introduction: Advances in machine vision and mobile electronics will be accelerated by the creation of sophisticated optoelectronic vision sensors that allow for sophisticated picture recognition of visual information and data pre-processing. Several new types of vision sensors have been devised in the last decade to solve these drawbacks, one of which is neuromorphic vision sensors, which have exciting qualities such as high temporal resolution, broad dynamic range, and low energy consumption. Neuromorphic sensors are inspired by the working principles of biological sensory neurons and would be useful in telemedicine, health surveillance, security monitoring, automatic driving, intelligent robots, and other applications of the Internet of Things.Methods: This paper provides a comprehensive review of various state-of-the-art AI vision sensors and frameworks.Results: The fundamental signal processing techniques deployed and the associated challenges were discussed.Discussion: Finally, the role of vision sensors in computer vision is also discussed.

show abstract

EyeTrAES: Fine-grained, Low-Latency Eye Tracking via Adaptive Event Slicing

Sen,

Bandara,

Gokarn

et al. 2024

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Eye-tracking technology has gained significant attention in recent years due to its wide range of applications in human-computer interaction, virtual and augmented reality, and wearable health. Traditional RGB camera-based eye-tracking systems often struggle with poor temporal resolution and computational constraints, limiting their effectiveness in capturing rapid eye movements. To address these limitations, we propose EyeTrAES, a novel approach using neuromorphic event cameras for high-fidelity tracking of natural pupillary movement that shows significant kinematic variance. One of EyeTrAES's highlights is the use of a novel adaptive windowing/slicing algorithm that ensures just the right amount of descriptive asynchronous event data accumulation within an event frame, across a wide range of eye movement patterns. EyeTrAES then applies lightweight image processing functions over accumulated event frames from just a single eye to perform pupil segmentation and tracking (as opposed to gaze-based techniques that require simultaneous tracking of both eyes). We show that these two techniques boost pupil tracking fidelity by 6+%, achieving IoU~=92%, while incurring at least 3x lower latency than competing pure event-based eye tracking alternatives [38]. We additionally demonstrate that the microscopic pupillary motion captured by EyeTrAES exhibits distinctive variations across individuals and can thus serve as a biometric fingerprint. For robust user authentication, we train a lightweight per-user Random Forest classifier using a novel feature vector of short-term pupillary kinematics, comprising a sliding window of pupil (location, velocity, acceleration) triples. Experimental studies with two different datasets (capturing eye movement across a range of environmental contexts) demonstrate that the EyeTrAES-based authentication technique can simultaneously achieve high authentication accuracy (~=0.82) and low processing latency (~=12ms), and significantly outperform multiple state-of-the-art competitive baselines.

show abstract

Adaptive Slicing Method of the Spatiotemporal Event Stream Obtained from a Dynamic Vision Sensor

Cited by 5 publications

References 26 publications

Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network

Event-Based Optical Flow Estimation with Spatio-Temporal Backpropagation Trained Spiking Neural Network

Artificial intelligence-based spatio-temporal vision sensors: applications and prospects

EyeTrAES: Fine-grained, Low-Latency Eye Tracking via Adaptive Event Slicing

Contact Info

Product

Resources

About