ALFO: Adaptive Light Field Over-Segmentation

Abstract

Automatic image over-segmentation into superpixels has attracted increasing attention from researchers to apply it as a pre-processing step for several computer vision applications. In 4D Light Field (LF) imaging, image over-segmentation aims at achieving not only superpixel compactness and accuracy but also cross-view consistency. Due to the high dimensionality of 4D LF images, depth information can be estimated and exploited during the over-segmentation along with spatial and visual appearance features. However, balancing between several hybrid features to generate robust superpixels for different 4D LF images is challenging and not adequately solved in existing solutions. In this paper, an automatic, adaptive, and view-consistent LF over-segmentation method based on normalized LF cues and K-means clustering is proposed. Initially, disparity maps for all LF views are estimated entirely to improve superpixel accuracy and consistency. Afterwards, by using K-means clustering, a 4D LF image is iteratively divided into regular superpixels that adhere to object boundaries and ensure cross-view consistency. Our proposed method can automatically adjust the clustering weights of the various features that characterize each superpixel based on the image content. Quantitative and qualitative results on several 4D LF datasets demonstrate outperforming performance of the proposed method in terms of superpixel accuracy, shape regularity and view consistency when using adaptive clustering weights, compared to the state-of-the-art 4D LF over-segmentation methods.INDEX TERMS Automatic segmentation, adaptive light field over-segmentation, superpixels.

View-consistent 4D Light Field Style Transfer using Neural Networks and Over-segmentation

“…After that, disparity maps for all input LF views are estimated using [12]; to ensure spatio-angular consistency during the propagation. Next, the 4D LF is over-segmented into spatioangular coherent regions (a.k.a superpixels), as in [13] to facilitate the propagation and respect object boundaries and occlusions. Afterwards, the stylization is propagated into all LF views through occlusion-aware back-projection from each view into all corner views.…”

“…LF over-segmentation is capable of adhering to object boundaries and creating a unique label for each homogenous region to facilitate subsequent editing tasks. In this paper, the recently proposed Adaptive LF Over-segmentation (ALFO) method [13] is used to guide the propagation in an occlusionaware manner. The ALFO method exploits color, disparity and position features to apply adaptive K-means clustering.…”

“…Additionally, it can robustly balance accuracy, shape regularity and view-consistency. In our experiments, the superpixel size is set to 20 as suggested in [13] as a reasonable size for robust adhesion to the borders.…”

“…where 𝑥 𝑖 (𝑢,𝑣) , 𝑦 𝑖 (𝑢,𝑣) are the spatial position coordinates of a pixel, 𝑖, which is located in a view of angular coordinates (𝑢, 𝑣), 𝑥 𝑖 𝑟𝑒𝑓 and 𝑦 𝑖 𝑟𝑒𝑓 are the spatial position in a reference view (i.e., 𝑟𝑒𝑓 in this paper represents a single corner view, hence, the same equation is applied for all corner views independently), and 𝑑 ℎ𝑜𝑟 (𝑢,𝑣)→𝑟𝑒𝑓 , 𝑑 𝑣𝑒𝑟 (𝑢,𝑣)→𝑟𝑒𝑓 are the horizontal and vertical disparity from view (𝑢, 𝑣) to the reference view. The used disparity estimation method [12] estimates disparity for adjacent views, therefore, for regularly sampled 4D LF views back-projection is applied by multiplying the disparity value by (𝑢 𝑟𝑒𝑓 − 𝑢), (𝑣 𝑟𝑒𝑓 − 𝑣) when computing 𝑥 𝑖 𝑟𝑒𝑓 and 𝑦 𝑖 𝑟𝑒𝑓 , respectively [13].…”

UrbanLF: A Comprehensive Light Field Dataset for Semantic Segmentation of Urban Scenes

“…Lv et al [80] build a hypergraph representation with LFSPs and present a method via graph-cut optimization. HAMAD et al [81] propose an automatic, adaptive, and view-consistent method based on normalized LF cues and K-means clustering.…”