Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks

Hagenaars, Jesse; Paredes-Vallés, Federico; Croon, Guido de

doi:10.48550/arxiv.2106.01862

Cited by 4 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…5. MVSEC comparison (dt = 4) of our method and two state-of-the-art baselines: ConvGRU-EV-FlowNet (USL) [21] and EV-FlowNet (SSL) [57]. For each sequence, the upper row shows the flow masked by the input events, and the lower row shows the IWE using the flow.…”

Section: Results On Mvsecmentioning

confidence: 99%

“…Current state-of-the-art approaches are ANNs [10,18,21,25,57,58], largely inspired by frame-based optical flow architectures [38,49]. Non-spiking-based approaches need to additionally adapt the input signal, converting the events into a tensor representation (event frames, time surfaces, voxel grids, etc.…”

Section: Prior Work On Event-based Optical Flow Estimationmentioning

confidence: 99%

“…Unsupervised methods rely solely on event data. Their loss function consists of an event alignment error using the flow predicted by the ANN [21,34,51,58]. Zhu et al [58] extended EV-FlowNet [57] to the unsupervised setting using a motion-compensation loss function in terms of average timestamp images.…”

Section: Prior Work On Event-based Optical Flow Estimationmentioning

confidence: 99%

“…Zhu et al [58] extended EV-FlowNet [57] to the unsupervised setting using a motion-compensation loss function in terms of average timestamp images. This approach has been used and improved in [21,34]. Paredes-Vallés et al [34] also proposed FireFlowNet, a lightweight ANN producing competitive results.…”

Section: Prior Work On Event-based Optical Flow Estimationmentioning

confidence: 99%

“…Preliminary work on applying CM to estimate optical flow has reported a problem of overfitting to the data, producing undesired flows that warp events to few pixels or lines [58] (i.e., event collapse [42]). This issue has been tackled by changing the objective function, from contrast to the energy of an average timestamp image [21,34,58], but this loss is not straightforward to interpret and makes training difficult to converge [13].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Secrets of Event-Based Optical Flow

Shiba¹,

Aoki²,

Gallego³

2022

Preprint

View full text Add to dashboard Cite

Event cameras respond to scene dynamics and offer advantages to estimate motion. Following recent image-based deep-learning achievements, optical flow estimation methods for event cameras have rushed to combine those image-based methods with event data. However, it requires several adaptations (data conversion, loss function, etc.) as they have very different properties. We develop a principled method to extend the Contrast Maximization framework to estimate optical flow from events alone. We investigate key elements: how to design the objective function to prevent overfitting, how to warp events to deal better with occlusions, and how to improve convergence with multi-scale raw events. With these key elements, our method ranks first among unsupervised methods on the MVSEC benchmark, and is competitive on the DSEC benchmark. Moreover, our method allows us to expose the issues of the ground truth flow in those benchmarks, and produces remarkable results when it is transferred to unsupervised learning settings. Our code is available at https://github.com/tub-rip/event based optical flow

show abstract

Section: Results On Mvsecmentioning

confidence: 99%

Section: Prior Work On Event-based Optical Flow Estimationmentioning

confidence: 99%

Section: Prior Work On Event-based Optical Flow Estimationmentioning

confidence: 99%

Section: Prior Work On Event-based Optical Flow Estimationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Secrets of Event-Based Optical Flow

Shiba¹,

Aoki²,

Gallego³

2022

Preprint

View full text Add to dashboard Cite

show abstract

Secrets of Event-Based Optical Flow

Shiba

Aoki

Gallego

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Direct learning-based deep spiking neural networks: a review

Guo

Huang

2023

Front. Neurosci.

View full text Add to dashboard Cite

The spiking neural network (SNN), as a promising brain-inspired computational model with binary spike information transmission mechanism, rich spatially-temporal dynamics, and event-driven characteristics, has received extensive attention. However, its intricately discontinuous spike mechanism brings difficulty to the optimization of the deep SNN. Since the surrogate gradient method can greatly mitigate the optimization difficulty and shows great potential in directly training deep SNNs, a variety of direct learning-based deep SNN works have been proposed and achieved satisfying progress in recent years. In this paper, we present a comprehensive survey of these direct learning-based deep SNN works, mainly categorized into accuracy improvement methods, efficiency improvement methods, and temporal dynamics utilization methods. In addition, we also divide these categorizations into finer granularities further to better organize and introduce them. Finally, the challenges and trends that may be faced in future research are prospected.

show abstract

Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks

Cited by 4 publications

References 0 publications

Secrets of Event-Based Optical Flow

Secrets of Event-Based Optical Flow

Secrets of Event-Based Optical Flow

Direct learning-based deep spiking neural networks: a review

Contact Info

Product

Resources

About