The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing

Argaw, Dawit Mureja; Heilbron, Fabian Caba; Lee, Joon-Young; Woodson, Markus; Kweon, In So

doi:10.1007/978-3-031-20074-8_12

Cited by 10 publications

(7 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The self-attention mechanism has been widely used in recent deep learning models due to its ability to handle long-range dependencies in computer vision tasks [23], [24]. Transformer models, which utilize self-attention, have emerged to solve various computer vision tasks such as image processing and pattern recognition [25], [26].…”

Section: B Learning With Attention Modelsmentioning

confidence: 99%

On the Combination of Multi-Input and Self-Attention for Sign Language Recognition

Hoai,

Van,

Anh

2024

IJACSA

View full text Add to dashboard Cite

Sign language recognition can be considered as a branch of human action recognition. The deaf-muted community utilizes upper body gestures to convey sign language words. With the rapid development of intelligent systems based on deep learning models, video-based sign language recognition models can be integrated into services and products to improve the quality of life for the deaf-muted community. However, comprehending the relationship between different words within videos is a complex and challenging task, particularly in understanding sign language actions in videos, further constraining the performance of previous methods. Recent methods have been explored to generate video annotations to address this challenge, such as creating questions and answers for images. An optimistic approach involves fine-tuning autoregressive language models trained using multiinput and self-attention mechanisms to facilitate understanding of sign language in videos. We have introduced a bidirectional transformer language model, MISA (multi-input self-attention), to enhance solutions for VideoQA (video question and answer) without relying on labeled annotations. Specifically, (1) one direction of the model generates descriptions for each frame of the video to learn from the frames and their descriptions, and (2) the other direction generates questions for each frame of the video, then integrates inference with the first aspect to produce questions that effectively identify sign language actions. Our proposed method has outperformed recent techniques in VideoQA by eliminating the need for manual labeling across various datasets, including CSL-Daily, PHOENIX14T, and PVSL (our dataset). Furthermore, it demonstrates competitive performance in lowdata environments and operates under supervision.

show abstract

Section: B Learning With Attention Modelsmentioning

confidence: 99%

On the Combination of Multi-Input and Self-Attention for Sign Language Recognition

Hoai,

Van,

Anh

2024

IJACSA

View full text Add to dashboard Cite

show abstract

“…It has been utilized several times in subsequent studies. In this paper, alongside MovieShots, we additionally incorporate another larger movie shot dataset, AVE [24], and select four shot attributes -shot motion, shot scale, shot angle, and shot type, for analysis.…”

Section: Movie Shot Datasetmentioning

confidence: 99%

“…In this paper, two large-scale movie shot datasets, MovieShots [1] and AVE [24], are utilized. MovieShots is the first large public movie shot dataset with complete shot sample clips constructed by the CUHK-SenseTime Joint Lab.…”

Section: Experiments 41 Datasetsmentioning

confidence: 99%

“…Following [24], we uniformly sample 16 frames from a shot clip as input. During the training process, we configure the batch size to be 8 and employ the mini-SGD optimizer with a momentum of 0.9.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…We validate our proposed method on two large movie shot datasets, MovieShots [1] and AVE [24], and compare the performance with all previous methods. Given the sample imbalance problems of movie shot datasets, we employ Macro-F1 for evaluation in addition to the regular Top-1 accuracy.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Toward Unified and Quantitative Cinematic Shot Attribute Analysis

Li,

Tian,

et al. 2023

Electronics

View full text Add to dashboard Cite

Cinematic Shot Attribute Analysis aims to analyze the intrinsic attributes of movie shots, such as movement and scale. In previous methods, specialized architectures were designed for each specific task and relied on the use of optical flow maps. In this paper, we consider shot attribute analysis as a unified task of motion–static weight allocation, and propose a motion–static dual-path architecture for recognizing various shot attributes. In this architecture, we design a new action cue generation module for adapting the end-to-end training process instead of a pre-trained optical flow network; and, to address the issue of limited samples in movie shot datasets, we design a fixed-size adjustment strategy to enable the network to directly utilize pre-trained vision transformer models while adapting to shot data inputs at arbitrary sample rates. In addition, we quantitatively analyze the sensitivity of different shot attributes to motion and static features for the first time. Subsequent experimental results on two datasets, MovieShots and AVE, demonstrate that our proposed method outperforms all previous approaches without increasing computational cost.

show abstract

Movie Lens: Discovering and Characterizing Editing Patterns in the Analysis of Short Movie Sequences

Vacchetti¹,

Cerquitelli²

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing

Cited by 10 publications

References 38 publications

On the Combination of Multi-Input and Self-Attention for Sign Language Recognition

On the Combination of Multi-Input and Self-Attention for Sign Language Recognition

Toward Unified and Quantitative Cinematic Shot Attribute Analysis

Movie Lens: Discovering and Characterizing Editing Patterns in the Analysis of Short Movie Sequences

Contact Info

Product

Resources

About