Real-Time High-Resolution Background Matting

Lin, Shanchuan; Ryabtsev, Andrey; Sengupta, Shamik; Curless, Brian; Seitz, Steve; Kemelmacher-Shlizerman, Ira

doi:10.48550/arxiv.2012.07810

Cited by 9 publications

(17 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the end, we have 56400 training images. For the test, we have three different test sets which are AIM [50], PhotoMatte85 (PM85) [32], and D646 [39]. We followed the same strategy and combined each person in the test set with 20 different images of the PASCAL VOC dataset [10].…”

Section: Resultsmentioning

confidence: 99%

“…We used Mean Squared Error (MSE), Mean Absolute Error (MAE), Sum of Absolute Difference (SAM), Gradient, and Connectivity metrics to evaluate our model as in the literature and we compared them with the previous works. For this, we chose available state-of-the-art methods, namely MOD-Net [26], BGM-V2 [32], FBA [12], in the literature and we tested them on the test sets in order to perform a fair comparison since different backgrounds may change the models' performances. Please note that we calculated these metrics over the whole image and MSE and MAE scores are scaled by 10 3 to improve the readability.…”

Section: Resultsmentioning

confidence: 99%

“…Besides, a oneframe delay trick is presented when the input data is a video sequence. In [32], a portrait matting system is designed to produce alpha matte and foreground images. The proposed method works in real-time with a high-resolution image.…”

Section: Image Mattingmentioning

confidence: 99%

“…Instead, one work employs original image and coarse annotated segmentation mask to generate a fine-grained alpha matte [43]. Moreover, recent work focuses on using original image and background image of the same image to produce alpha matte without using any other information such as coarse alpha matte and trimap [32], while another works utilizes only the input image to achieve fine-grained alpha matte [35,26].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Alpha Matte Generation from Single Input for Portrait Matting

Yaman¹,

Ekenel²,

Waibel³

2021

Preprint

View full text Add to dashboard Cite

Portrait matting is an important research problem with a wide range of applications, such as video conference app, image/video editing, and post-production. The goal is to predict an alpha matte that identifies the effect of each pixel on the foreground subject. Traditional approaches and most of the existing works utilized an additional input, e.g., trimap, background image, to predict alpha matte. However, providing additional input is not always practical. Besides, models are too sensitive to these additional inputs. In this paper, we introduce an additional input-free approach to perform portrait matting using Generative Adversarial Nets (GANs). We divide the main task into two subtasks. For this, we propose a segmentation network for the person segmentation and the alpha generation network for alpha matte prediction. While the segmentation network takes an input image and produces a coarse segmentation map, the alpha generation network utilizes the same input image as well as a coarse segmentation map that is produced by the segmentation network to predict the alpha matte. Besides, we present a segmentation encoding block to downsample the coarse segmentation map and provide feature representation to the residual block. Furthermore, we propose border loss to penalize only the borders of the subject separately which is more likely to be challenging and we also adapt perceptual loss for portrait matting. To train the proposed system, we combine two different popular training datasets to improve the amount of data as well as diversity to address domain shift problems in the inference time. We tested our model on three different benchmark datasets, namely Adobe Image Matting dataset, Portrait Matting dataset, and Distinctions dataset. The proposed method outperformed the MODNet method that also takes a single input. Besides, we obtained comparable results with BGM-V2 and FBA methods that require additional input.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Image Mattingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Alpha Matte Generation from Single Input for Portrait Matting

Yaman¹,

Ekenel²,

Waibel³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Chen et al [3]propose to automatic generate trimap using semantic information. Sengupta et al [15] and Lin et al [10] propose to take another photo of the background as auxiliary input. Wei et al [17] propose to use user clicks as foreground and background hints.…”

Section: Trimap-free Matting Methodsmentioning

confidence: 99%

Deep Image Matting with Flexible Guidance Input

Cheng¹,

Xu²,

Jiang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Image matting is an important computer vision problem. Many existing matting methods require a hand-made trimap to provide auxiliary information, which is very expensive and limits the real world usage. Recently, some trimap-free methods have been proposed, which completely get rid of any user input. However, their performance lag far behind trimap-based methods due to the lack of guidance information. In this paper, we propose a matting method that use Flexible Guidance Input as user hint, which means our method can use trimap, scribblemap or clickmap as guidance information or even work without any guidance input. To achieve this, we propose Progressive Trimap Deformation(PTD) scheme that gradually shrink the area of the foreground and background of the trimap with the training step increases and finally become a scribblemap. To make our network robust to any user scribble and click, we randomly sample points on foreground and background and perform curve fitting. Moreover, we propose Semantic Fusion Module(SFM) which utilize the Feature Pyramid Enhancement Module(FPEM) and Joint Pyramid Upsampling(JPU) in matting task for the first time. The experiments show that our method can achieve state-of-the-art results comparing with existing trimap-based and trimap-free methods. Our demo is at https://github.com/Charch-630/FGI-Matting.

show abstract

Rethinking Portrait Matting with Privacy Preserving

Zhang

et al. 2023

Int J Comput Vis

View full text Add to dashboard Cite

Recently, there has been an increasing concern about the privacy issue raised by identifiable information in machine learning. However, previous portrait matting methods were all based on identifiable images. To fill the gap, we present P3M-10k, which is the first large-scale anonymized benchmark for Privacy-Preserving Portrait Matting (P3M). P3M-10k consists of 10,421 high resolution face-blurred portrait images along with high-quality alpha mattes, which enables us to systematically evaluate both trimap-free and trimap-based matting methods and obtain some useful findings about model generalization ability under the privacy preserving training (PPT) setting. We also present a unified matting model dubbed P3M-Net that is compatible with both CNN and transformer backbones. To further mitigate the cross-domain performance gap issue under the PPT setting, we devise a simple yet effective Copy and Paste strategy (P3M-CP), which borrows facial information from public celebrity images and directs the network to reacquire the face context at both data and feature level. Extensive experiments on P3M-10k and public benchmarks demonstrate the superiority of P3M-Net over state-of-the-art methods and the effectiveness of P3M-CP in improving the cross-domain generalization ability, implying a great significance of P3M for future research and real-world applications. The dataset, code and models are available here (https://github.com/ViTAE-Transformer/P3M-Net).

show abstract

Real-Time High-Resolution Background Matting

Cited by 9 publications

References 32 publications

Alpha Matte Generation from Single Input for Portrait Matting

Alpha Matte Generation from Single Input for Portrait Matting

Deep Image Matting with Flexible Guidance Input

Rethinking Portrait Matting with Privacy Preserving

Contact Info

Product

Resources

About