Using a Single Input to Forecast Human Action Keystates in Everyday Pick and Place Actions

Abstract

We define action keystates as the start or end of an action that contains information such as the human pose and time. Existing methods that forecast the human pose use recurrent networks that input and output a sequence of poses. In this paper, we present a method tailored for everyday pick and place actions where the object of interest is known. In contrast to existing methods, ours uses an input from a single timestep to directly forecast (i) the key pose the instant the pick or place action is performed and (ii) the time it takes to get to the predicted key pose. Experimental results show that our method outperforms the state-of-the-art for key pose forecasting and is comparable for time forecasting while running at least an order of magnitude faster. Further ablative studies reveal the significance of the object of interest in enabling the total number of parameters across all existing methods to be reduced by at least 90% without any degradation in performance. a

Action-Conditioned Generation of Bimanual Object Manipulation Sequences

“…Early deterministic models tend to use recurrent networks such as Gated Recurrent Units (GRU) (Chung et al 2014) or fully convolutional layers. Several other works incorporate additional context such as eye gaze (Razali and Demiris 2021) or the object coordinates (Razali and Demiris 2022;Taheri et al 2022). Lastly, the context-aware model (Corona et al 2020) forecasts both the human pose and object motion and is related to our work, although there exist several notable differences.…”

Forecasting Bimanual Object Manipulation Sequences from Unimanual Observations

“…These works can also be categorized into deterministic (Martinez, Black, and Romero 2017) or stochastic (Liu et al 2021), using Variational Autoencoders (VAEs) (Kingma and Welling 2013) or Generative Adversarial Networks (GANs) (Goodfellow et al 2014) respectively, with the design choice hinging on whether there is sufficient variation to be learnt by the model. Many recent works incorporate additional context such as scene (Corona et al 2020), eye gaze (Razali and Demiris 2022b;Zheng et al 2022), or object coordinates (Razali and Demiris 2022a).…”