Video-based Human-Object Interaction Detection from Tubelet Tokens

Abstract

We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatiotemporal representations, for video-based human-object interaction (V-HOI) detection. The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens along spatial and temporal domains, which enjoy two benefits: 1) Compactness: each tubelet token is learned by a selective attention mechanism to reduce redundant spatial dependencies from others; 2) Expressiveness: each tubelet token is enabled to align with a semantic instance, i.e., an object or a human, across frames, thanks to agglomeration and linking. The effectiveness and efficiency of TUTOR are verified by extensive experiments. Results shows our method outperforms existing works by large margins, with a relative mAP gain of 16.14% on VidHOI and a 2 points gain on CAD-120 as well as a 4× speedup.Preprint. Under review.

Jointly Modeling Spatio-Temporal Features of Tactile Signals for Action Classification

“…Tubelet Inputs As the spatial and temporal dimensions of the tactile signals can be redundant, directly adopting the whole data in classification may result in reduced efficiency. Motivated by previous video transformer models that convert the video clip into tubelets to alleviate the spatiotemporal redundancy, we follow these studies by transferring the tactile signals into a tubelet sequence (Arnab et al 2021b;Liu et al 2021;Fan et al 2021;Tu et al 2022). We define a tubelet as Q ∈ R L×P ×P , where L represents its sequence length (i.e., the number of frames) and P represents the patch size (i.e., height and width).…”