Occlusion Detection for Automatic Video Editing

Liao, Junhua; Duan, Haihan; Li, Xin; Xu, Haoran; Yang, Yanbing; Cai, Wei; Chen, Yanru; Chen, Liangyin

doi:10.1145/3394171.3413725

Cited by 13 publications

(3 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Besides, intelligent creation tools are in great demand as they can help users efficiently create customized dynamic content, e.g., video, animation [27,28]. Some researchers focus on several key steps, such as frame composition [60], shot selection [23,25], shot cut suggestion [35]. Others tackle high-level automatic procedures with simple user interactions and take multiple videos captured by different cameras to produce a coherent video in different application scenarios [1,24,48] using different data sources [5,31,39,54].…”

Section: Related Workmentioning

confidence: 99%

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

Rao¹,

Jiang²,

Guo³

et al. 2023

Preprint

View full text Add to dashboard Cite

Dynamic Storyboard 2 Dynamic Storyboard 1 Virtual Environment 📷 Camera Scripts 2: Dolly Full High-angle 📷 Camera scripts 1: Push Medium Eye-level 📖 Story scripts: Jane and Jack are arguing in living room Figure 1. We present Virtual Dynamic Storyboard (VDS) that takes user input story and camera scripts and automatically composes dynamic storyboards in an engine-based virtual environment for pre-visualization. Here we show two results produced by VDS with top-ranked scores of quality. Video demos can be found in the supplementary.

show abstract

Section: Related Workmentioning

confidence: 99%

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

Rao¹,

Jiang²,

Guo³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The size of the "patch" proposed in ViT [12] greatly increases the "receptive field" of the network. There have been many successful Transformer-based applications in computer vision, such as image classification [13][14][15], video detection [16,17], and action recognition [18,19]. Therefore, it is very meaningful to explore Transformer-based dualmodality action recognition methods.…”

Section: Introductionmentioning

confidence: 99%

A Novel Two-Stream Transformer-Based Framework for Multi-Modality Human Action Recognition

et al. 2023

Self Cite

View full text Add to dashboard Cite

Due to the great success of Vision Transformer (ViT) in image classification tasks, many pure Transformer architectures for human action recognition have been proposed. However, very few works have attempted to use Transformer to conduct bimodal action recognition, i.e., both skeleton and RGB modalities for action recognition. As proved in many previous works, RGB modality and skeleton modality are complementary to each other in human action recognition tasks. How to use both RGB and skeleton modalities for action recognition in a Transformer-based framework is a challenge. In this paper, we propose RGBSformer, a novel two-stream pure Transformer-based framework for human action recognition using both RGB and skeleton modalities. Using only RGB videos, we can acquire skeleton data and generate corresponding skeleton heatmaps. Then, we input skeleton heatmaps and RGB frames to Transformer at different temporal and spatial resolutions. Because the skeleton heatmaps are primary features compared to the original RGB frames, we use fewer attention layers in the skeleton stream. At the same time, two ways are proposed to fuse the information of two streams. Experiments demonstrate that the proposed framework achieves the state of the art on four benchmarks: three widely used datasets, Kinetics400, NTU RGB+D 60, and NTU RGB+D 120, and the fine-grained dataset FineGym99.

show abstract

“…Active speaker detection is a multi-modal task aiming to identify active speakers from a set of candidates in an arbitrary video. This task plays an essential role in speaker diarization [7,41], speaker tracking [27,28], automatic video editing [10,19], and other applications, which has attracted extensive attention from both industry and academia. Figure 1.…”

Section: Introductionmentioning

confidence: 99%

A Light Weight Model for Active Speaker Detection

Liao¹,

Duan²,

Kanghui³

et al. 2023

Preprint

View full text Add to dashboard Cite

Active speaker detection is a challenging task in audiovisual scenario understanding, which aims to detect who is speaking in one or more speakers scenarios. This task has received extensive attention as it is crucial in applications such as speaker diarization, speaker tracking, and automatic video editing. The existing studies try to improve performance by inputting multiple candidate information and designing complex models. Although these methods achieved outstanding performance, their high consumption of memory and computational power make them difficult to be applied in resource-limited scenarios. Therefore, we construct a lightweight active speaker detection architecture by reducing input candidates, splitting 2D and 3D convolutions for audio-visual feature extraction, and applying gated recurrent unit (GRU) with low computational complexity for cross-modal modeling. Experimental results on the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP performance (94.1% vs. 94.2%), while the resource costs are significantly lower than the state-of-the-art method, especially in model parameters (1.0M vs. 22.5M, about 23×) and FLOPs (0.6G vs. 2.6G, about 4×). In addition, our framework also performs well on the Columbia dataset showing good robustness. The code and model weights are available at https: //github.com/Junhua-Liao/Light-ASD.

show abstract

Occlusion Detection for Automatic Video Editing

Cited by 13 publications

References 47 publications

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production

A Novel Two-Stream Transformer-Based Framework for Multi-Modality Human Action Recognition

A Light Weight Model for Active Speaker Detection

Contact Info

Product

Resources

About