Mask4D: 4D Convolution Network for Light Field Occlusion Removal

Abstract

Current light field (LF) occlusion removal approaches usually select only a part of sub-aperture images (SAIs) or simply stack all SAIs to reconstruct the center view, which destroys the spatial layout of SAIs. In this paper, we present a simple yet effective LF occlusion removal method name Mask4D, which is a 4D convolution-based encoder-decoder network. We propose to keep the spatial layout of SAIs and construct all SAIs as a 5D input tensor to fully exploit the spatial connection information between SAIs. In particular, except for center view reconstruction, we jointly predict the occlusion mask to disentangle the occlusion mask from the occluded content. Extensive evaluations demonstrate that our Mask4D surpasses the state-of-the-art approaches across different datasets. Moreover, visualizations show that Mask4D predicts the occlusion mask precisely and the reconstructed center view looks more realistic than other approaches. Our code will be publicly available.

Disentangling Light Fields for Super-Resolution and Disparity Estimation

“…Apart from these aforementioned networks, many architectures have also been developed for LF image processing, such as bi-directional recurrent structure [41], 4D convolution [12], [55], [56] and cost volumes [28], [68]. The key difference between existing schemes and our disentangling mechanism is, we can fully use the information from all angular views and incorporate the LF structure prior.…”

“…4D Convolution. Since several recent works [12], [55], [56] used 4D convolutions to handle LF data and have achieved promising performance, we compare our disentangling mechanism with 4D convolutions by replacing our Distg-Block with a series of 4D residual blocks. As shown in Table 1, stacking 4D convolutions can result in a very large model size (i.e., 22.3M for 2×SR) but cannot introduce performance improvements.…”

“…L IGHT field (LF) cameras can capture both intensity and directions of light rays, and record 3D geometry in a convenient and efficient manner. By encoding 3D scene cues into 4D LF images (i.e., 2D for spatial dimension and 2D for angular dimension), LF cameras enable many attractive applications such as post-capture refocusing [1], [2], depth sensing [3]- [6], reflectance estimation [7]- [10], and foreground de-occlusion [11], [12]. With recent development of LF imaging devices (e.g., plenoptic cameras [13], coded aperture cameras [14], moving gantries [15], and camera arrays [16]), LF image processing has drawn extensive research interests due to its large potential in both academic and industrial communities [17].…”

Light Field Image Super-Resolution Using Deep Residual Networks on Lenslet Images

“…Unlike conventional imaging, which captures the 2D projection of light rays, LF imaging collects data with many dimensions [ 1 ]. This abundance of visual information in LF pictures, in addition to their immersive description of the real world, may help several image processing and computer vision tasks, such as depth estimation [ 2 , 3 ], de-occlusion [ 4 , 5 ], salient object detection [ 6 , 7 ], and image post-refocus [ 8 ].…”

RCA-LF: Dense Light Field Reconstruction Using Residual Channel Attention Networks

“…These images contain spatial and angular information about the 3D scenes. As a result, many applications have developed and benefited greatly from this huge amount of information, such as de-occlusion [ 1 , 2 ], depth-sensing [ 3 , 4 , 5 ], saliency detection [ 6 ], and salient object detection [ 7 ]. In addition, LF could be promising to ease other applications such as the fruit-picking robot, where a robot traverses a whole field and harvests on its own [ 8 , 9 ].…”