LiFF: Light Field Features in Scale and Depth

Abstract

Feature detectors and descriptors are key low-level vision tools that many higher-level tasks build on. Unfortunately these fail in the presence of challenging light transport effects including partial occlusion, low contrast, and reflective or refractive surfaces. Building on spatio-angular imaging modalities offered by emerging light field cameras, we introduce a new and computationally efficient 4D light field feature detector and descriptor: LiFF. LiFF is scale invariant and utilizes the full 4D light field to detect features that are robust to changes in perspective. This is particularly useful for structure from motion (SfM) and other tasks that match features across viewpoints of a scene. We demonstrate significantly improved 3D reconstructions via SfM when using LiFF instead of the leading 2D or 4D features, and show that LiFF runs an order of magnitude faster than the leading 4D approach. Finally, LiFF inherently estimates depth for each feature, opening a path for future research in light field-based SfM.

Dense Light Field Coding: A Survey

“…It is also possible to combine these techniques to meet some specific requirements for spatial and angular density. For instance, in [55], a gantry structure with a lenslet camera is used for capturing LFs with a baseline varying between micrometers (inside the lenslet camera) to meters (by varying the camera poses). Additionally, in [12], a one dimensional (1D) array of GoPro cameras in a vertical arc are placed in a horizontally rotating gantry structure to capture dense LFs with 360 • FoV.…”

All-in-depth via Cross-baseline Light Field Camera

“…To the best of our knowledge, we are the first work to bind stereo and light-field for large-scale depth range estimation, although some works use more than one light-field camera for other purposes. For example, Dansereau et al [12] directly extract feature information on light-field based 4D model using 3 light-field cameras. Wang et al [32] attached an additional standard camera to a LF camera for LF video using a learning-based approach.…”

“…The difference form ESPCN is using 3D LF data as input of network (The input size is W × H × MN, where W and H are width and height of sub-aperture view in LF, and M, N are the number of two angular dimensions). As shown in 5, we implement an experiment to show the superiority of M-ESPCN, notice to be fair, the training data for both ESPCN and M-ESPCN comes from same dataset [12]. Using M-ESPCN, complementary information in sub-aperture views of LF is used in the EPI domain, which leads to less noise and better performance than naive up-sample methods like bicubic or up-sample sub-aperture views independently using ESPCN.…”

Implementation of a Depth from Light Field Algorithm on FPGA

“…Also some studies focused on using the technology in space applications [60,61]. The light field features in scale space and depth proposed in [62] can also be included in this category of algorithms.…”