When Video Classification Meets Incremental Classes

Abstract

With the rapid development of social media, tremendous videos with new classes are generated daily, which raise an urgent demand for video classification methods that can continuously update new classes while maintaining the knowledge of old videos with limited storage and computing resources. In this paper, we summarize this task as Class-Incremental Video Classification (CIVC) and propose a novel framework to address it. As a subarea of incremental learning tasks, the challenge of catastrophic forgetting is unavoidable in CIVC.To better alleviate it, we utilize some characteristics of videos. First, we decompose the spatio-temporal knowledge before distillation rather than treating it as a whole in the knowledge transfer process; trajectory is also used to refine the decomposition. Second, we propose a dual granularity exemplar selection method to select and store representative video instances of old classes and key-frames inside videos under a tight storage budget. We benchmark our method and previous SOTA class-incremental learning methods on Something-Something V2 and Kinetics datasets, and our method outperforms previous methods significantly. CCS CONCEPTS• Computing methodologies → Activity recognition and understanding.

Recent Advances of Continual Learning in Computer Vision: An Overview

“…They thus proposed a method named Distillation and Retrospection to obtain a better balance between preservation of previous tasks and adaptation to new tasks. In video classification, Zhao et al [186] adapted the knowledge distillation based method by separately distilling the spatial and temporal knowledge. They also introduced a dual granularity exemplar selection method to store only key frames of representative video instances from the previous tasks.…”

vCLIMB: A Novel Video Class Incremental Learning Benchmark

“…Despite the growing interest in CL in the image domain, the first three works to report results on video data were only recently published. Zhao et al [44] proposed a spatio-temporal knowledge transfer strategy to mitigate catastrophic forgetting. A concurrent work [26] estimated the subset of feature channels that contributes the most to the predictions of the previous tasks, Although existing works bootstrapped the study of continual learning in video data, all of them use different evaluation protocols, making direct comparisons between methods difficult.…”

“…With such large volumes of data, it is important to develop models that can effectively learn from continuous streams of untrimmed video data. Remarkably, few research efforts have addressed continual learning with video [24,26,44]. Despite these current works, video continual learning methods still show a large variability in their experimental protocols, making direct comparisons hard to establish.…”

MGH: Metadata Guided Hypergraph Modeling for Unsupervised Person Re-identification

“…In the computer vision community, metric learning [19,29,42,59] is widely used for learning discriminative feature representations.…”