EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

Abstract

Learning coordinate-based volumetric 3D scene representations such as neural radiance fields (NeRF) has been so far studied assuming RGB or RGB-D images as inputs. At the same time, it is known from the neuroscience literature that human visual system (HVS) is tailored to process asynchronous brightness changes rather than synchronous RGB images, in order to build and continuously update mental 3D representations of the surroundings for navigation and survival. Visual sensors that were inspired by HVS principles are event cameras. Thus, events are sparse and asynchronous per-pixel brightness (or colour channel) change signals. In contrast to existing works on neural 3D scene representation learning, this paper approaches the problem from a new perspective. We demonstrate that it is possible to learn NeRF suitable for novel-view synthesis in the RGB space from asynchronous event streams. Our models achieve high visual accuracy of the rendered novel views of challenging scenes in the RGB space, even though they are trained with substantially fewer data (i.e., event streams from a single event camera moving around the object) and more efficiently (due to the inherent sparsity of event streams) than the existing NeRF models trained with RGB images. We will release our datasets and the source code, see https: //4dqv.mpi-inf.mpg.de/EventNeRF/.

HKUST-Landslide Susceptibility Dataset (HKUST-LSD): A benchmark dataset for landslide susceptibility assessment in Hong Kong

“…Such a characteristic has motivated us to approach the task of event-to-video reconstruction by optimizing the INR of video from its temporal derivatives. Previous research has investigated the potential of INR for novel view synthesis using event data [20,24,38]. These approaches reconstruct 3D neural radiance fields using multiple event sequences with known camera poses from a stationary scene.…”

Pan‐risk factor for a comprehensive cardiovascular health management

“…The vast body of existing research in NeRF was investigated based on RGB-based cameras, which suffer from inevitable shortcomings, e.g., low dynamic range and motion blur in unfavourable visual conditions. Thus, recent attention has been paid to the usage of event cameras for NeRF [235], [236], [237]. The first work is EventNeRF [235], which is trained with pure event-based supervision.…”

“…Thus, recent attention has been paid to the usage of event cameras for NeRF [235], [236], [237]. The first work is EventNeRF [235], which is trained with pure event-based supervision. It demonstrates that the NeRF estimation from a single fastmoving event camera in unfavourable scenarios (e.g., fast-moving objects, motion blur, or insufficient lighting) is feasible while frame-based approaches fail.…”