Label-Efficient Online Continual Object Detection in Streaming Video

Abstract

To thrive in evolving environments, humans are capable of continual acquisition and transfer of new knowledge, from a continuous video stream, with minimal supervisions, while retaining previously learnt experiences. In contrast to human learning, most standard continual learning benchmarks focus on learning from static iid images in fully supervised settings. Here, we examine a more realistic and challenging problem-Label-Efficient Online Continual Object Detection (LEOCOD) in video streams. By addressing this problem, it would greatly benefit many real-world applications with reduced data annotation costs and model retraining time. To tackle this problem, we seek inspirations from complementary learning systems (CLS) in human brains and propose a computational model, dubbed as Efficient-CLS. Functionally correlated with the hippocampus and the neocortex in CLS, Efficient-CLS posits a memory encoding mechanism involving bidirectional interaction between fast and slow learners via synaptic weight transfers and pattern replays. We test Efficient-CLS and competitive baselines in two challenging real-world video stream datasets. Like humans, Efficient-CLS learns to detect new object classes incrementally from a continuous temporal stream of non-repeating video with minimal forgetting. Remarkably, with only 25% annotated video frames, our Efficient-CLS still leads among all comparative models, which are trained with 100% annotations on all video frames. The data and source code will be publicly available at https://github.com/showlab/Efficient-CLS.

Managing Metaverse Data Tsunami: Actionable Insights

“…This is a major roadblock for developing more advanced pretrained models, e.g. continual learning and update of the pretrained model given the newly arrived data stream [181] in the ever-changing metaverse. To overcome this challenge, it is important to optimize data communication between storage and compute nodes, and to exploit parameter-efficient continual learning and transfer learning of large deep learning models.…”

Comparative Study of Natural Replay and Experience Replay in Online Object Detection

“…Datasets for OOD OAK [28] stands as the pioneering benchmark for OOD, offering ego-centric video snippets captured from a student's perspective at Krishna campus. Notably, OAK has been used in the context of semisupervised OOD [29] as well as the EgoObjects dataset.…”

“…Evaluating the forgetting rate of an algorithm on past knowledge becomes challenging due to the presence of NR, which acts as an important parameter determining the classes that will be forgotten in a given scenario. Existing benchmarks for OOD do not explicitly define or quantify NR [28,29], and this lack of consideration hinders a precise assessment of forgetting.…”