Shot Contrastive Self-Supervised Learning for Scene Boundary Detection

Chen, Shixing; Nie, Xiaohan; Fan, David D.; Zhang, Dongqing; Bhat, Vimal; Hamid, Raffay

doi:10.1109/cvpr46437.2021.00967

Cited by 46 publications

(39 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, there are various works on contrastive video representation learning, which viewed contrastive learning from temporal [33], spatio-temporal [35], and temporal-equivariant [20] perspectives. Especially, [6] applied self-supervised contrastive pretext task to learn appropriate feature representation, proving its effectiveness on detecting shot frames. However, it still needs downstream task training (supervised learning) to get final results, while our approach directly yields event boundaries.…”

Section: Contrastive Representation Learningmentioning

confidence: 99%

UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection

Kang

Kim

et al. 2021

Preprint

View full text Add to dashboard Cite

Generic Event Boundary Detection (GEBD) is a newly suggested video understanding task that aims to find one level deeper semantic boundaries of events. Bridging the gap between natural human perception and video understanding, it has various potential applications, including interpretable and semantically valid video parsing. Still at an early development stage, existing GEBD solvers are simple extensions of relevant video understanding tasks, disregarding GEBD's distinctive characteristics. In this paper, we propose a novel framework for unsupervised/supervised GEBD, by using the Temporal Self-similarity Matrix (TSM) as the video representation. The new Recursive TSM Parsing (RTP) algorithm exploits local diagonal patterns in TSM to detect boundaries, and it is combined with the Boundary Contrastive (BoCo) loss to train our encoder to generate more informative TSMs. Our framework can be applied to both unsupervised and supervised settings, with both achieving state-of-the-art performance by a huge margin in GEBD benchmark. Especially, our unsupervised method outperforms the previous state-of-the-art "supervised" model, implying its exceptional efficacy.

show abstract

Section: Contrastive Representation Learningmentioning

confidence: 99%

UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection

Kang

Kim

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Temporal segmentation of actions in videos has been widely explored in previous works [40,41,69,86,89,104]. Video shot boundary detection and scene detection tasks are also relevant and has been explored in many previous studies [11,29,30,65,102], which aim at finding the visual change or scene boundaries.…”

Section: Related Workmentioning

confidence: 99%

“…The objective of optimal transport involves solving linear programming and may cause potential computational burdens since it has 𝑂 (𝑛 3 ) efficiency. To solve this issue, we add an entropic regularization term equation (11) and the objective of our optimal transport distance becomes…”

Section: Cross-domain Alignment For Multimodal Summarizationmentioning

confidence: 99%

MHMS: Multimodal Hierarchical Multimedia Summarization

Jielin¹,

Zhu²,

Xu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multimedia summarization with multimodal output can play an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. In this work, we propose a multimodal hierarchical multimedia summarization (MHMS) framework by interacting visual and language domains to generate both video and textual summaries. Our MHMS method contains video and textual segmentation and summarization module, respectively. It formulates a cross-domain alignment objective with optimal transport distance which leverages cross-domain interaction to generate the representative keyframe and textual summary. We evaluated MHMS on three recent multimodal datasets and demonstrated the effectiveness of our method in producing high-quality multimodal summaries.

show abstract

“…Video scene detection is the most relevant task. However, previous methods only used visual information to detect the scene change [52,56,57,11,81], so the methods can not be adopted directly for Livestream videos either.…”

Section: Related Workmentioning

confidence: 99%

LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos

Jielin¹,

Dernoncourt²,

Bui³

et al. 2022

Preprint

View full text Add to dashboard Cite

Livestream videos have become a significant part of online learning, where design, digital marketing, creative painting, and other skills are taught by experienced experts in the sessions, making them valuable materials. However, Livestream tutorial videos are usually hours long, recorded, and uploaded to the Internet directly after the live sessions, making it hard for other people to catch up quickly. An outline will be a beneficial solution, which requires the video to be temporally segmented according to topics. In this work, we introduced a large Livestream video dataset named Mul-tiLive, and formulated the temporal segmentation of the long Livestream videos (TSLLV) task. We propose LiveSeg, an unsupervised Livestream video temporal Segmentation solution, which takes advantage of multimodal features from different domains. Our method achieved a 16.8% F1score performance improvement compared with the stateof-the-art method.

show abstract

Shot Contrastive Self-Supervised Learning for Scene Boundary Detection

Cited by 46 publications

References 23 publications

UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection

UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection

MHMS: Multimodal Hierarchical Multimedia Summarization

LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos

Contact Info

Product

Resources

About