Watch-n-Patch: Unsupervised Learning of Actions and Relations

Abstract

There is a large variation in the activities that humans perform in their everyday lives. We consider modeling these composite human activities which comprises multiple basic level actions in a completely unsupervised setting. Our model learns high-level co-occurrence and temporal relations between the actions. We consider the video as a sequence of short-term action clips, which contains human-words and object-words. An activity is about a set of action-topics and object-topics indicating which actions are present and which objects are interacting with. We then propose a new probabilistic model relating the words and the topics. It allows us to model long-range action relations that commonly exist in the composite activities, which is challenging in previous works. We apply our model to the unsupervised action segmentation and clustering, and to a novel application that detects forgotten actions, which we call action patching. For evaluation, we contribute a new challenging RGB-D activity video dataset recorded by the new Kinect v2, which contains several human daily activities as compositions of multiple actions interacting with different objects. Moreover, we develop a robotic system that watches and reminds people using our action patching algorithm. Our robotic setup can be easily deployed on any assistive robots.

Unsupervised Long-Term Routine Modelling Using Dynamic Bayesian Networks

“…The recording is not long-term as their research focus is on analysing multiple individuals performing the same task. The morning dataset from [18] and Watch-n-Patch dataset [19] both contain no repetitive activities per person, and it is mostly scripted, which do not meet the requirement for routine discovery.…”

Recognition of Activities of Daily Living via Hierarchical Long-Short Term Memory Networks

“…LDA models the probability that a video (document) is generated from a set of actions (topics) where each action (topic) is a distribution of words from a codebook. They are employed for activity recognition [37] and forgotten action detection [38].…”

“…We quantitatively compare our hierarchical LSTM model (H-LSTM) with state-of-the-art approaches on the Watchn-Patch dataset. To the best of our knowledge the existing methods evaluated on this dataset are: Hidden Markov Model (HMM) [3], Latent Dirichlet Allocation (LDA) [4], Causal Topic Model (CaTM) [37] and Watch-Bot Topic Model (WBTM) [38]. All results are reported from [38].…”

“…To the best of our knowledge the existing methods evaluated on this dataset are: Hidden Markov Model (HMM) [3], Latent Dirichlet Allocation (LDA) [4], Causal Topic Model (CaTM) [37] and Watch-Bot Topic Model (WBTM) [38]. All results are reported from [38]. For our method, we use a fixed-length segmentation with a window size W S = 15 and 500 units of the LSTM cell.…”

Benchmark for Kitchen20, a daily life dataset for audio-based human action recognition

“…Robots which act and perceive in a human environment must rely on embedded sensors of different modalities in order to perceive and recognize human activities, and in order to provide help if necessary. Vision is often used as the principal modality in performing such recognition tasks [12,17] but audio can contribute and help in these challenging situations by providing additional information. Moreover, a robot does not always have a clear line of sight, and could thus rely on audio to help understand what the human is currently doing.…”