MobileFusion: Real-Time Volumetric Surface Reconstruction and Dense Tracking on Mobile Phones

Abstract

We present the first pipeline for real-time volumetric surface reconstruction and dense 6DoF camera tracking running purely on standard, off-the-shelf mobile phones. Using only the embedded RGB camera, our system allows users to scan objects of varying shape, size, and appearance in seconds, with real-time feedback during the capture process. Unlike existing state of the art methods, which produce only point-based 3D models on the phone, or require cloud-based processing, our hybrid GPU/CPU pipeline is unique in that it creates a connected 3D surface model directly on the device at 25Hz. In each frame, we perform dense 6DoF tracking, which continuously registers the RGB input to the incrementally built 3D model, minimizing a noise aware photoconsistency error metric. This is followed by efficient key-frame selection, and dense per-frame stereo matching. These depth maps are fused volumetrically using a method akin to KinectFusion, producing compelling surface models. For each frame, the implicit surface is extracted for live user feedback and pose estimation. We demonstrate scans of a variety of objects, and compare to a Kinect-based baseline, showing on average ∼ 1.5cm error. We qualitatively compare to a state of the art point-based mobile phone method, demonstrating an order of magnitude faster scanning times, and fully connected surface models.

Visual SLAM algorithms: a survey from 2010 to 2016

“…The DTAM algorithm is optimized for achieving realtime processing on mobile phones [46]. Basically, these methods [43,46,47] are designed for fast and online 3D modeling.…”

“…Basically, these methods [43,46,47] are designed for fast and online 3D modeling. It should be noted that Stühmer et al previously proposed a variational approach for estimating depth information for every pixels [47].…”

Object detection with BRIEF descriptors and locality sensitive matching for augmented reality

“…Artırılmış gerçeklik benzeri uygulamalarda sıkça kullanılırlar. Son yıllarda hem mobil işlemcilerin yaygınlaşması hem de işlem kapasitelerinin artması nedeniyle, mobil nesne arama yöntemleri de geliştirilmiştir [1].…”

Binocular mobile augmented reality based on stereo camera tracking

“…Although this approach was well implemented on a smartphone with 3-D display, it was still not suitable for processing in real time, because of its expensive computational costs. Recently, it has become possible to run natural feature-based planar object tracking [4] as well as 3-D object tracking in real time even on a mobile platform [5,6] with the development of hardware and software. Nevertheless, expansion of those methods to stereo camera tracking requires the careful design of an algorithm and optimization, because of the additional computation required for consistent registration across cameras.…”