Temporal Upsampling of Depth Maps Using a Hybrid Camera

Yuan, Mingze; Gao, Lin; Fu, Hongbo; Xia, Shihong

doi:10.1109/tvcg.2018.2812879

Cited by 16 publications

(12 citation statements)

References 61 publications

(114 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This premise is especially true when fast-moving human subjects are involved, such as in sport assistance and virtual reality. Monocular solutions involving RGB-D sensors [59] and high-speed cameras [26] have been explored, although they cannot meet the computational requirements of real-time applications. On the other hand, event-cameras achieve high recording speed without saturating bandwidth and resources.…”

Section: Related Workmentioning

confidence: 99%

Lifting Monocular Events to 3D Human Poses

Scarpellini¹,

Morerio²,

Bue³

2021

Preprint

View full text Add to dashboard Cite

This paper presents a novel 3D human pose estimation approach using a single stream of asynchronous events as input. Most of the state-of-the-art approaches solve this task with RGB cameras, however struggling when subjects are moving fast. On the other hand, event-based 3D pose estimation benefits from the advantages of event-cameras, especially their efficiency and robustness to appearance changes. Yet, finding human poses in asynchronous events is in general more challenging than standard RGB pose estimation, since little or no events are triggered in static scenes. Here we propose the first learning-based method for 3D human pose from a single stream of events. Our method consists of two steps. First, we process the event-camera stream to predict three orthogonal heatmaps per joint; each heatmap is the projection of of the joint onto one orthogonal plane. Next, we fuse the sets of heatmaps to estimate 3D localisation of the body joints. As a further contribution, we make available a new, challenging dataset for eventbased human pose estimation by simulating events from the RGB Human3.6m dataset. Experiments demonstrate that our method achieves solid accuracy, narrowing the performance gap between standard RGB and event-based vision. The code is freely available at https://iit-pavis. github.io/lifting_events_to_3d_hpe.

show abstract

Section: Related Workmentioning

confidence: 99%

Lifting Monocular Events to 3D Human Poses

Scarpellini¹,

Morerio²,

Bue³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Yuan, M. details a framework for temporal upsampling using a hybrid camera [11]. This approach achieves high accuracy, but it was shown to be unable to handle rapid scene-changes, such as those occurring in a typical head-mounted display scenario when a user quickly turns his or her head.…”

Section: Related Workmentioning

confidence: 99%

Accelerating Real-Time, High-Resolution Depth Upsampling on FPGAs

Langerman

Sabogal

Ramesh

et al. 2018

2018 IEEE International Conference on Image Processing, Applications and Systems (IPAS)

View full text Add to dashboard Cite

While the popularity of high-resolution, computer-vision applications (e.g. mixed reality, autonomous vehicles) is increasing, there have been complementary advances in time-of-flight (ToF) depth-sensor resolution and quality. These advances in ToF sensors provide a platform that can enable real-time, depth-upsampling algorithms targeted for high-resolution video systems with low-latency requirements. This thesis demonstrates that filter-based upsampling algorithms are feasible for real-time, low-power scenarios, such as those on HMDs. Specifically, the author profiled, parallelized, and accelerated a filter-based depth-upsampling algorithm on an FPGA using high-level synthesis tools from Xilinx. We show that our accelerated algorithm can accurately upsample the resolution and reduce the noise of ToF sensors. We also demonstrate that this algorithm exceeds the real-time requirements of 90 frames-per-second (FPS) and 11 ms latency of mixed-reality hardware, achieving a lower-bound speedup of 40 times over the fastest CPU-only version and a 4.7 times speedup over the original GPU implementation [1]. v

show abstract

“…Several recent works [30,71] revealed the importance of the high frame rate camera systems for tracking fast motions. However, they still suffer from the aforementioned fundamental problem -the high frame rate leads to excessive amounts of raw data and large bandwidth requirement for data processing (e.g., capturing RGB stream of VGA resolution at 1000 fps from a single view for one minute yields 51.5GB of data).…”

Section: Introductionmentioning

confidence: 99%

“…However, they still suffer from the aforementioned fundamental problem -the high frame rate leads to excessive amounts of raw data and large bandwidth requirement for data processing (e.g., capturing RGB stream of VGA resolution at 1000 fps from a single view for one minute yields 51.5GB of data). Moreover, both methods [30,71] assume 1) well-lit scenarios for compensating the short exposure time at high frame rate, and 2) indoor capture due to the limitation of the IR-based depth sensor.…”

Section: Introductionmentioning

confidence: 99%

EventCap: Monocular 3D Capture of High-Speed Human Motions Using an Event Camera

Golyanik

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

The high frame rate is a critical requirement for capturing fast human motions. In this setting, existing markerless image-based methods are constrained by the lighting requirement, the high data bandwidth and the consequent high computation overhead. In this paper, we propose EventCap -the first approach for 3D capturing of high-speed human motions using a single event camera. Our method combines model-based optimization and CNNbased human pose detection to capture high-frequency motion details and to reduce the drifting in the tracking. As a result, we can capture fast motions at millisecond resolution with significantly higher data efficiency than using high frame rate videos. Experiments on our new event-based fast human motion dataset demonstrate the effectiveness and accuracy of our method, as well as its robustness to challenging lighting conditions.

show abstract

Temporal Upsampling of Depth Maps Using a Hybrid Camera

Cited by 16 publications

References 61 publications

Lifting Monocular Events to 3D Human Poses

Lifting Monocular Events to 3D Human Poses

Accelerating Real-Time, High-Resolution Depth Upsampling on FPGAs

EventCap: Monocular 3D Capture of High-Speed Human Motions Using an Event Camera

Contact Info

Product

Resources

About