Learning Stereo Depth Estimation with Bio-Inspired Spike Cameras

Wang, Yixuan; Li, Jianing; Zhu, Lin; Xiang, Xijie; Huang, Tiejun

doi:10.1109/icme52920.2022.9859975

Cited by 12 publications

(5 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SCFlow (Hu et al 2022) estimates optical flow directly from the binary spike streams based on a pyramidal network. SSDEFormer (Wang et al 2022) and Spike Transformer (Zhang et al 2022) estimate binocular and monocular depth for spike cameras with transformers, respectively.…”

Section: Related Workmentioning

confidence: 99%

“…Inter-Moment Hierarchical Fusion As shown in the top of Fig. 4, different from previous works that fuse temporal information of spikes in a single operation (Hu et al 2022;Wang et al 2022), we retain the time information in the feature extraction procedure. The hierarchical fusion strategy is inspired by video restoration tasks (Maggioni et al 2021;Isobe et al 2020;Xu et al 2021a;Chan et al 2022; Liu et al 2022) that process a video frame with adjacent frames as references.…”

Section: Hierarchical Spatial-temporal Fusion Modulementioning

confidence: 99%

“…(3) Multi-window representation. SSDEFormer (Wang et al 2022) uses the multi-window temporal average of the spike stream for representation. The window size varies from 1 to T .…”

Section: Ablations Studiesmentioning

confidence: 99%

“…Compared with event cameras, spike cameras can better recover the scene, especially for regions with fewer textures and motion. Many pixel-level tasks are researched for spike cameras, such as reconstruction (Zhao et al 2021b;Zheng et al 2021;Zhao et al 2021c), optical flow (Hu et al 2022) and depth estimation (Wang et al 2022;Zhang et al 2022).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

Zhao,

Xiong,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

As an emerging neuromorphic camera with an asynchronous working mechanism, spike camera shows good potential for high-speed vision tasks. Each pixel in spike camera accumulates photons persistently and fires a spike whenever the accumulation exceeds a threshold. Such high-frequency fine-granularity photon recording facilitates the analysis and recovery of dynamic scenes with high-speed motion. This paper considers the optical flow estimation problem for spike cameras. Due to the Poisson nature of incoming photons, the occurrence of spikes is random and fluctuating, making conventional image matching inefficient. We propose a Hierarchical Spatial-Temporal (HiST) fusion module for spike representation to pursue reliable feature matching and develop a robust optical flow network, dubbed as HiST-SFlow. The HiST extracts features at multiple moments and hierarchically fuses the spatial-temporal information. We also propose an intra-moment filtering module to further extract the feature and suppress the influence of randomness in spikes. A scene loss is proposed to ensure that this hierarchical representation recovers the essential visual information in the scene. Experimental results demonstrate that the proposed method achieves state-of-the-art performance compared with the existing methods. The source codes are available at https://github.com/ruizhao26/HiST-SFlow.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Hierarchical Spatial-temporal Fusion Modulementioning

confidence: 99%

“…(3) Multi-window representation. SSDEFormer (Wang et al 2022) uses the multi-window temporal average of the spike stream for representation. The window size varies from 1 to T .…”

Section: Ablations Studiesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

Zhao,

Xiong,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Spike cameras possess several advantages, including high temporal resolution (Zheng et al 2023b), high dynamic range, and rich preservation of spatial texture. These advantages have led to wide applications in various downstream tasks, such as optical flow estimation (Hu et al 2022;Zhao et al 2022;, object tracking (Zheng et al 2023a), and depth estimation (Zhang et al 2022a;Wang et al 2022). Among these tasks, the reconstruction task (Zhang et al 2023) serves as the fundamental basis.…”

Section: Spike-based Image Reconstructionmentioning

confidence: 99%

Transient Glimpses: Unveiling Occluded Backgrounds through the Spike Camera

Zhang,

Chen,

Zheng

et al. 2024

AAAI

View full text Add to dashboard Cite

The de-occlusion problem, involving extracting clear background images by removing foreground occlusions, holds significant practical importance but poses considerable challenges. Most current research predominantly focuses on generating discrete images from calibrated camera arrays, but this approach often struggles with dense occlusions and fast motions due to limited perspectives and motion blur. To overcome these limitations, an effective solution requires the integration of multi-view visual information. The spike camera, as an innovative neuromorphic sensor, shows promise with its ultra-high temporal resolution and dynamic range. In this study, we propose a novel approach that utilizes a single spike camera for continuous multi-view imaging to address occlusion removal. By rapidly moving the spike camera, we capture a dense stream of spikes from occluded scenes. Our model, SpkOccNet, processes these spikes by integrating multi-view spatial-temporal information via long-short-window feature extractor (LSW) and employs a novel cross-view mutual attention-based module (CVA) for effective fusion and refinement. Additionally, to facilitate research in occlusion removal, we introduce the S-OCC dataset, which consists of real-world spike-based data. Experimental results demonstrate the efficiency and generalization capabilities of our model in effectively removing dense occlusions across diverse scenes. Public project page: https://github.com/Leozhangjiyuan/SpikeDeOcclusion.

show abstract

SpikeGS: Learning 3D Gaussian Fields from Continuous Spike Stream

Yu,

Peng,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Learning Stereo Depth Estimation with Bio-Inspired Spike Cameras

Cited by 12 publications

References 13 publications

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

Optical Flow for Spike Camera with Hierarchical Spatial-Temporal Spike Fusion

Transient Glimpses: Unveiling Occluded Backgrounds through the Spike Camera

SpikeGS: Learning 3D Gaussian Fields from Continuous Spike Stream

Contact Info

Product

Resources

About