Self-Supervised Learning to Detect Key Frames in Videos

Gilani, Syed Zulqarnain; Feng, Mingtao; Zhang, Liang; Qin, Hanlin; Mian, Ajmal

doi:10.3390/s20236941

Cited by 36 publications

(27 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because the direct comparison between key frame detection and our approach was not fair, we introduces related research in this section. In one recently reported study, Yan et al [6] provided a new method that required no annotation. Their proposed approach had a selfsupervised learning framework that could identify the key frames in a video.…”

Section: Key Frame Detectionmentioning

confidence: 99%

See 1 more Smart Citation

Extending Contrastive Learning to Unsupervised Redundancy Identification

Jung

Kim

2022

Applied Sciences

View full text Add to dashboard Cite

Modern deep neural network (DNN)-based approaches have delivered great performance for computer vision tasks; however, they require a massive annotation cost due to their data-hungry nature. Hence, given a fixed budget and unlabeled examples, improving the quality of examples to be annotated is a clever step to obtain good generalization of DNN. One of key issues that could hurt the quality of examples is the presence of redundancy, in which the most examples exhibit similar visual context (e.g., same background). Redundant examples barely contribute to the performance but rather require additional annotation cost. Hence, prior to the annotation process, identifying redundancy is a key step to avoid unnecessary cost. In this work, we proved that the coreset score based on cosine similarity (cossim) is effective for identifying redundant examples. This is because the collective magnitude of the gradient over redundant examples exhibits a large value compared to the others. As a result, contrastive learning first attempts to reduce the loss of redundancy. Consequently, cossim for the redundancy set exhibited a high value (low coreset score). We first viewed the redundancy identification as the gradient magnitude. In this way, we effectively removed redundant examples from two datasets (KITTI, BDD10K), resulting in a better performance in terms of detection and semantic segmentation.

show abstract

Section: Key Frame Detectionmentioning

confidence: 99%

“…The most related topic could be key frame detection, which aims to discover the frame that provides powerful features from a video. Yan et al [6] recently achieved key frame detection in a self-supervised manner, that is, at zero annotation cost. They verified their method by applying a thorough experiment on an action recognition dataset.…”

Section: Introductionmentioning

confidence: 99%

Extending Contrastive Learning to Unsupervised Redundancy Identification

Jung

Kim

2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Additionally, the work in (Breck et al, 2019) presents a schema to detect errors in data pipelines for ML solutions. However, video-based data processing pipelines pose additional challenges related to key-frame identification (Yan et al, 2020) and efficient data storage and retrieval (Kang et al, 2019). In this work, we present end-to-end ML pipelines that are capable of ingesting large volumes of video data streams, identifying the key-frames that need annotation, and creating an efficient storage-retrieval schema that aid data collection for subsequent modeling cycles.…”

Section: Related Workmentioning

confidence: 99%

Video-Data Pipelines for Machine Learning Applications

Roychowdhury¹,

Sato²

2021

Preprint

View full text Add to dashboard Cite

Data pipelines are an essential component for end-to-end solutions that take machine learning algorithms to production. Engineering data pipelines for video-sequences poses several challenges including isolation of key-frames from video sequences that are high quality and represent significant variations in the scene. Manual isolation of such quality key-frames can take hours of sifting through hours' worth of video data. In this work, we present a data pipeline framework that can automate this process of manual frame sifting in video sequences by controlling the fraction of frames that can be removed based on image quality and content type. Additionally, the frames that are retained can be automatically tagged per sequence, thereby simplifying the process of automated data retrieval for future ML model deployments. We analyze the performance of the proposed video-data pipeline for versioned deployment and monitoring for object detection algorithms that are trained on outdoor autonomous driving video sequences. The proposed video-data pipeline can retain anywhere between 0.1-20% of the all input frames that are representative of high image quality and high variations in content. This frame selection, automated scene tagging followed by model verification can be completed in under 30 seconds for 22 video-sequences under analysis in this work. Thus, the proposed framework can be scaled to additional video-sequence data sets for automating ML versioned deployments.

show abstract

“…Computer Vision and scene understanding applications rely on processing large volumes of video data to learn the patterns related to the objects/regions of interest (ROIs) [1]. Therefore, automated video sequence processing systems for efficient frame tagging, storage and retrieval become key components for building such automated machine vision systems.…”

Section: Introductionmentioning

confidence: 99%

Semi-supervised and Deep learning Frameworks for Video Classification and Key-frame Identification

Roychowdhury¹

2022

Preprint

View full text Add to dashboard Cite

Automating video-based data and machine learning pipelines poses several challenges including metadata generation for efficient storage and retrieval and isolation of key-frames for scene understanding tasks. In this work, we present two semi-supervised approaches that automate this process of manual frame sifting in video streams by automatically classifying scenes for content and filtering frames for fine-tuning scene understanding tasks. The first rule-based method starts from a pre-trained object detector and it assigns scene type, uncertainty and lighting categories to each frame based on probability distributions of foreground objects. Next, frames with the highest uncertainty and structural dissimilarity are isolated as key-frames. The second method relies on the simCLR model for frame encoding followed by label-spreading from 20% of frame samples to label the remaining frames for scene and lighting categories. Also, clustering the video frames in the encoded feature space further isolates key-frames at cluster boundaries. The proposed methods achieve 64-93% accuracy for automated scene categorization for outdoor image videos from public domain datasets of JAAD and KITTI. Also, less than 10% of all input frames can be filtered as key-frames that can then be sent for annotation and fine tuning of machine vision algorithms. Thus, the proposed framework can be scaled to additional video data streams for automated training of perception-driven systems with minimal training images.

show abstract

Self-Supervised Learning to Detect Key Frames in Videos

Cited by 36 publications

References 66 publications

Extending Contrastive Learning to Unsupervised Redundancy Identification

Extending Contrastive Learning to Unsupervised Redundancy Identification

Video-Data Pipelines for Machine Learning Applications

Semi-supervised and Deep learning Frameworks for Video Classification and Key-frame Identification

Contact Info

Product

Resources

About