Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding

Abstract

An event happening in the world is often made of different activities and actions that can unfold simultaneously or sequentially within a few seconds. However, most large-scale datasets built to train models for action recognition provide a single label per video clip. Consequently, models can be incorrectly penalized for classifying actions that exist in the videos but are not explicitly labeled and do not learn the full spectrum of information that would be mandatory to more completely comprehend different events and eventually learn causality between them. Towards this goal, we augmented the existing video dataset, Moments in Time (MiT), to include over two million action labels for over one million three second videos. This multi-label dataset introduces novel challenges on how to train and analyze models for multi-action detection. Here, we present baseline results for multi-action recognition using loss functions adapted for long tail multi-label learning and provide improved methods for visualizing and interpreting models trained for multi-label action detection.

AR-Net: Adaptive Frame Resolution for Efficient Action Recognition

“…Efficient Action Recognition. Action recognition has made rapid progress with the introduction of a number of large-scale datasets such as Kinetics [7] and Moments-In-Time [39,40]. Early methods have studied action recognition using shallow classification models such as SVM on top of local visual features extracted from a video [34,53].…”

Video Action Understanding

“…[36,83,325]. Unlike these surveys, we will use a broader definition of action, one that includes actions of both human and non-human actors because (1) video datasets are being introduced that use this broader definition [177,178], (2) most deep learning metrics and methods are equally applicable to both settings, and (3) the colloquial use of action has no distinction between human and non-human actors. Merriam-Webster's Dictionary and the Oxford English Dictionary define action as "an act done" and "something done or performed", respectively [174,197].…”

“…Each single-instance video lasts for 3 seconds. The dataset was improved to Multi-Moments in Time (M-MiT) [178] in 2019 by increasing the number of videos to 1.02 million, pruning vague classes, and increasing the number of labels per video (2.01 million total labels). MiT and M-MiT are interesting benchmarks because of the focus on inter-class and intra-class variation.…”

“…From there, precision up to rank 𝑘, interpolated 𝐴𝑃 for a particular class, and 𝑚𝐴𝑃 are calculated as shown in Equations 2, 3, and 4. This metric is used for MultiTHUMOS [301], ActivityNet 1.3 [21] when applied as an untrimmed AR problem, and Multi-Moments in Time [178]. One possible variant of multi-label 𝑚𝐴𝑃 involves only computing 𝐴𝑃 for each class up to a specified rank 𝑘.…”

Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions

“…These sequential events present additional challenges for video datasets where single annotations may not be sufficient to explain the events depicted. Multi-label approaches to video annotation have attempted to address this problem by labeling multiple actions in a video [47,22,73]. However, these methods focus on single domain annotations, such as actions or objects, and do not capture additional contextual information, such as "person angrily putting down the dirty glass on a rusted table", which can change the interpretation of an event and how it fits into a sequence of observations.…”