EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

Rudnev, Viktor; Golyanik, Vladislav; Wang, Jiayi; Seidel, Hans‐Peter; Mueller, Franziska; Elgharib, Mohamed; Theobalt, Christian

doi:10.1109/iccv48922.2021.01216

Cited by 38 publications

(56 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Novel View Synthesis from Event Streams. Unique properties of event cameras (i.e., low latency, ultra-low power consumption, high dynamic range and no motion blur) motivated their usage in Simultaneous Localisation and Mapping (SLAM) [48,19,35] and 3D reconstruction of hands [38,30] and human bodies [49,58]. Several event-based SLAM methods [19,35] rely on explicit feature matching in the event space and reconstruct sparse 3D models of environments from a single event stream.…”

Section: Related Workmentioning

confidence: 99%

“…The analysis-by-synthesis method of Nevhi et al [30] operates on events only and supports arbitrary non-rigid objects but still requires a 3D mesh of the target object. EventHands [38] regresses sparse 3D hand keypoints and, thus, cannot be used for novel view synthesis of dense hand surfaces. In stark contrast to these works, we learn-for the first time-an implicit 3D scene representation from event streams that enables photo-realistic novel view synthesis in the RGB space.…”

Section: Related Workmentioning

confidence: 99%

“…Using a single event camera instead of an RGB camera have several merits, including its ability to handle strong motion blur that exist in RGB images and could cause several RGB-based computer vision methods to fail. Event cameras, on the other hand, do not suffer from motion blur and have already brought valuable contributions in the important and long-standing problems of hand [38] and body [49] tracking in this regards. Event cameras also have a higher dynamic range compared to RGB cameras and, thus, are able to handle dark scenes, which is also a fundamental limitation of traditional intensitybased methods.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

Rudnev¹,

Elgharib²,

Theobalt³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Learning coordinate-based volumetric 3D scene representations such as neural radiance fields (NeRF) has been so far studied assuming RGB or RGB-D images as inputs. At the same time, it is known from the neuroscience literature that human visual system (HVS) is tailored to process asynchronous brightness changes rather than synchronous RGB images, in order to build and continuously update mental 3D representations of the surroundings for navigation and survival. Visual sensors that were inspired by HVS principles are event cameras. Thus, events are sparse and asynchronous per-pixel brightness (or colour channel) change signals. In contrast to existing works on neural 3D scene representation learning, this paper approaches the problem from a new perspective. We demonstrate that it is possible to learn NeRF suitable for novel-view synthesis in the RGB space from asynchronous event streams. Our models achieve high visual accuracy of the rendered novel views of challenging scenes in the RGB space, even though they are trained with substantially fewer data (i.e., event streams from a single event camera moving around the object) and more efficiently (due to the inherent sparsity of event streams) than the existing NeRF models trained with RGB images. We will release our datasets and the source code, see https: //4dqv.mpi-inf.mpg.de/EventNeRF/.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

Rudnev¹,

Elgharib²,

Theobalt³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…EventCap of Xu et al [29] is a hybrid approach that relies on deblurred greyscale images recorded at usual frame rates and an event stream from DAVIS240C to track a rigged 3D model of an actor. The recent work of Rudnev et al [22] proposes EventHands, i.e., a neural method for 3D hand reconstruction from a single event stream. In contrast to EventHands, our method does not require large corpora of training data, and works for general non-rigid objects with hands only being one example object.…”

Section: Related Workmentioning

confidence: 99%

Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

Nehvi¹,

Golyanik²,

Mueller³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based tracking and reconstruction of non-rigid objects in 3D, like hands and body, has been either tackled using explicit event trajectories or large-scale datasets. In contrast, our method does not require any such processing or data, and can be readily applied to incoming event streams. We show the effectiveness of our approach for various types of non-rigid objects and compare to existing methods for non-rigid 3D tracking. In our experiments, the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors. The source code and the new data are publicly available 1 .

show abstract

“…Neural Hand Pose and Shape Estimation. Most approaches for 3D hand reconstruction and tracking from monocular RGB, depth and event cameras estimate hand poses only, i.e., a sparse set of 3D hand joints [1], [2], [3], [21], [22], [23]. The approaches which estimate hand shape and pose simultaneously are still in the minority.…”

Section: Related Workmentioning

confidence: 99%

HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks

Malik¹,

Shimada²,

Elhayek³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

3D hand shape and pose estimation from a single depth map is a new and challenging computer vision problem with many applications. Existing methods addressing it directly regress hand meshes via 2D convolutional neural networks, which leads to artifacts due to perspective distortions in the images. To address the limitations of the existing methods, we develop HandVoxNet++, i.e., a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner. The input to our network is a 3D voxelized-depth-map-based on the truncated signed distance function (TSDF). HandVoxNet++ relies on two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology and which is the most accurate representation. The second representation is the hand surface that preserves the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which does not rely on training data. In extensive evaluations on three public benchmarks, i.e., SynHand5M, depth-based HANDS19 challenge and HO-3D, the proposed HandVoxNet++ achieves the state-of-the-art performance. In this journal extension of our previous approach presented at CVPR 2020, we gain 41.09% and 13.7% higher shape alignment accuracy on SynHand5M and HANDS19 datasets, respectively. Our method is ranked first on the HANDS19 challenge dataset (Task 1: Depth-Based 3D Hand Pose Estimation) at the moment of the submission of our results to the portal in August 2020.

show abstract

EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

Cited by 38 publications

References 61 publications

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks

Contact Info

Product

Resources

About