Robust High-Resolution Video Matting with Temporal Guidance

Lin, Shanchuan; Yang, Linjie; Saleemi, Imran; Sengupta, Shamik

doi:10.1109/wacv51458.2022.00319

Cited by 113 publications

(61 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We categorize matting methods to three categories: trimap-based methods, trimap-free methods, and background-based methods. Therefore, we compare the proposed method with all three kinds methods, FBA [9], RVM [20] and RTBGM [19]. FBA is the representative of trimap-based methods.…”

Section: Evaluation Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Adaptive Background Matting Using Background Matching

Liu¹

2022

Preprint

View full text Add to dashboard Cite

Due to the difficulty of solving the matting problem, lots of methods use some kinds of assistance to acquire high quality alpha matte. Green screen matting methods rely on physical equipment. Trimap-based methods take manual interactions as external input. Background-based methods require a pre-captured, static background. The methods are not flexible and convenient enough to use widely. Trimapfree methods are flexible but not stable in complicated video applications. To be stable and flexible in real applications, we propose an adaptive background matting method. The user first captures their videos freely, moving the cameras. Then the user captures the background video afterwards, roughly covering the previous captured regions. We use dynamic background video instead of static background for accurate matting. The proposed method is convenient to use in any scenes as the static camera and background is no more the limitation. To achieve this goal, we use background matching network to find the best-matched background frame by frame from dynamic backgrounds. Then, robust semantic estimation network is used to estimate the coarse alpha matte. Finally, we crop and zoom the target region according to the coarse alpha matte, and estimate the final accurate alpha matte. In experiments, the proposed method is able to perform comparably against the state-ofthe-art matting methods.

show abstract

Section: Evaluation Resultsmentioning

confidence: 99%

“…The running time is listed in Table 1. FBA [9] takes 377ms, RVM [20] takes about 12.1ms, RTBGM [19] takes 24.1ms. For our method, the matting time (REN+AEN) is 34.8ms, which is also very fast.…”

Section: Evaluation Resultsmentioning

confidence: 99%

Adaptive Background Matting Using Background Matching

Liu¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Tasks and Datasets of image segmentation are closely related in deep learning era. Some of the segmentation tasks like [14,24,54,55,67,83,94,105], are even directly built upon the datasets. Their problem formulations are exactly the same: P = F (θ, I), where I and P are the input image and the binary map output, respectively.…”

Section: Related Workmentioning

confidence: 99%

“…Currently, deep models are the most popular solutions for most of the segmentation tasks. Many different deep architectures have been proposed to achieve better performance, such as FCNbased [60] feature aggregation models [9,42,62,93,99,110,111,117], Encoder-Decoder architectures [3,10,77,81], Coarse-to-Fine (or Predict-Refine) models [13,18,55,78,90,95,96], Vision Transformers [58,118], etc. Besides, many real-time models [27,44,51,70,71,107,114] are developed to balance the performance and time costs.…”

Section: Existing Modelsmentioning

confidence: 99%

“…However, existing image segmentation tasks mainly focus on segmenting objects with specific characteristics, e.g., salient [90,94,109], camouflaged [26,48,85], meticulous [54,105] or specific categories [45,55,67,81,83]. Most of them have the same input/output formats, and barely use exclusive mechanisms designed for segmenting targets in their models, which means almost all tasks are dataset-dependent.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Highly Accurate Dichotomous Image Segmentation

Qin¹,

Dai²,

Hu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Deep Learning Model to Classify and Monitor Idiopathic Scoliosis in Adolescents Using a Single Smartphone Photograph

Zhang,

Zhu,

Zhao

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceAdolescent idiopathic scoliosis (AIS) is the most common pediatric spinal disorder. Routine physical examinations by trained personnel are critical to diagnose severity and monitor curve progression in AIS. In the presence of concerning malformation, radiographs are necessary for diagnosis or follow-up, guiding further management, such as bracing correction for moderate malformation and spine surgery for severe malformation. If left unattended, progressive deterioration occurs in two-thirds of patients, leading to significant health concerns for growing children.ObjectiveTo assess the ability of an open platform application (app) using a validated deep learning model to classify AIS severity and curve type, as well as identify progression.Design, Setting, and ParticipantsThis diagnostic study was performed with data from radiographs and smartphone photographs of the backs of adolescent patients at spine clinics. The ScolioNets deep learning model was developed and validated in a prospective training cohort, then incorporated and tested in the AlignProCARE open platform app in 2022. Ground truths (GTs) included severity, curve type, and progression as manually annotated by 2 experienced spine specialists based on the radiographic examinations of the participants’ spines. The GTs and app results were blindly compared with another 2 spine surgeons’ assessments of unclothed back appearance. Data were analyzed from October 2022 to February 2023.ExposureAcquisitions of unclothed back photographs using a mobile app.Main Outcomes and MeasuresOutcomes of interest were classification of AIS severity and progression. Quantitative statistical analyses were performed to assess the performance of the deep learning model in classifying the deformity as well as in distinguishing progression during 6-month follow-up.ResultsThe training data set consisted of 1780 patients (1295 [72.8%] female; mean [SD] age, 14.3 [3.3] years), and the prospective testing data sets consisted of 378 patients (279 [73.8%] female; mean [SD] age, 14.3 [3.8] years) and 376 follow-ups (294 [78.2%] female; mean [SD] age, 15.6 [2.9] years). The model recommended follow-up with an area under receiver operating characteristic curve (AUC) of 0.839 (95% CI, 0.789-0.882) and considering surgery with an AUC of 0.902 (95% CI, 0.859-0.936), while showing good ability to distinguish among thoracic (AUC, 0.777 [95% CI, 0.745-0.808]), thoracolumbar or lumbar (AUC, 0.760 [95% CI, 0.727-0.791]), or mixed (AUC, 0.860 [95% CI, 0.834-0.887]) curve types. For follow-ups, the model distinguished participants with or without curve progression with an AUC of 0.757 (95% CI, 0.630-0.858). Compared with both surgeons, the model could recognize severities and curve types with a higher sensitivity (eg, sensitivity for recommending follow-up: model, 84.88% [95% CI, 75.54%-91.70%]; senior surgeon, 44.19%; junior surgeon, 62.79%) and negative predictive values (NPVs; eg, NPV for recommending follow-up: model, 89.22% [95% CI, 84.25%-93.70%]; senior surgeon, 71.76%; junior surgeon, 79.35%). For distinguishing curve progression, the sensitivity and NPV were comparable with the senior surgeons (sensitivity, 63.33% [95% CI, 43.86%-80.87%] vs 77.42%; NPV, 68.57% [95% CI, 56.78%-78.37%] vs 72.00%). The junior surgeon reported an inability to identify curve types and progression by observing the unclothed back alone.ConclusionsThis diagnostic study of adolescent patients screened for AIS found that the deep learning app had the potential for out-of-hospital accessible and radiation-free management of children with scoliosis, with comparable performance as spine surgeons experienced in AIS management.

show abstract

Robust High-Resolution Video Matting with Temporal Guidance

Cited by 113 publications

References 28 publications

Adaptive Background Matting Using Background Matching

Adaptive Background Matting Using Background Matching

Highly Accurate Dichotomous Image Segmentation

Deep Learning Model to Classify and Monitor Idiopathic Scoliosis in Adolescents Using a Single Smartphone Photograph

Contact Info

Product

Resources

About