Beyond Action Recognition: Action Completion in RGB-D Data

Abstract

Robust motion representations for action recognition have achieved remarkable performance in both controlled and 'in-the-wild' scenarios. Such representations are primarily assessed for their ability to label a sequence according to some predefined action classes (e.g. walk, wave, open). Although increasingly accurate, these classifiers are likely to label a sequence, even if the action has not been fully completed, because the motion observed is similar enough to the training set. Consider the case where one attempts to drink but realises the beverage is too hot. A drinking-vs-all classifier is likely to recognise this action as drinking regardless. We introduce the term action completion as a step beyond the task of action recognition. It aims to recognise whether the action's goal has been successfully achieved. The notion of completion differs per action and could be infeasible to verify using a visual sensor, however, for many actions, an observer would be able to make the distinction by noticing subtle differences in motion.We address incompletion in a supervised approach, using a new dataset that contains 414 complete as well as incomplete sequences, captured using a depth sensor, spanning 6 actions (switch, plug, open, pull, pick and drink). For each action, we varied the conditions so the action cannot be completed. For example, for plug, subjects were given a plug that does not match the socket, while for pull, a drawer was locked so could not be pulled, and similarly for the other actions. Given labelled complete and incomplete sequences of the same action, we build a model of completion of that action as a binary classifier for each of our actions.Since the notion of completion differs per action, a general action completion method should investigate the performance of different types of features to accommodate the various action classes. For example, for the action pick, the difference between complete and incomplete actions originates from the subtle change in body pose when holding an object, or by observing an object in the hand. On the other hand, for the action drink, the speed at which the action is performed is better able to assess the completion. We propose a method that chooses the feature(s) suitable for recognising completion from a pool of depth features using 'leave-oneperson-out' cross validation on the training set and automatically selecting the most discriminative feature(s). Figure 1: For a complete drink (green) and an incomplete drink (blue) sequences from our dataset, both are classified as drink when using drink vs. plug classifier (a). The proposed supervised action completion model (b) identifies the incomplete sequence.

Am I Done? Predicting Action Progress in Videos

“…In [36] an LSTM is used to obtain a temporally increasing confidence to discriminate between similar actions in early stages while Aliakbarian et al [2] combine action and context aware features to make predictions when a very small percentage of the action is observed. In [18] RGB-D data is used to discriminate between complete and incomplete actions and [52] tries to detect missing sub activities to timely remind them to the user. More recently, Heidarivincheh et al [19] is able to predict the frame's relative position of the completion moment by either classification or regression.…”

Weakly-Supervised Completion Moment Detection using Temporal Attention

“…Although these works present a fine-grained analysis from the action progression, they also consider complete attempts and do not detect or localise the completion moment. Action completion [13] differs from these works, as it focuses on the action's goal. In [14], completion moment detection was addressed using a classification-regression network which outputs frame-level predictions.…”

“…Action completion was first introduced in [13] to assess whether the action's goal is achieved. The approach outputs sequence-level predictions of completion to distinguish complete sequences from incomplete ones.…”

“…We show that our proposed approach outperforms [14] when fully supervised, Frame-level preand post-completion labels Sequence-level complete label Figure 1. While frame-level labels (left) are used in a fully-supervised approach like [13], we detect the completion moment only using weak labels, i.e. sequence level complete and incomplete (right).…”

“…We evaluate our approach on selected actions from HMDB [17], UCF101 [33] and RGBD-AC [13]. We show that learning temporal attention decreases the completion detection error, i.e.…”