MILD: Multimodal Interactive Latent Dynamics for Learning Human-Robot Interaction

Prasad, V.; Koert, Dorothea; Stock‐Homburg, Ruth; Peters, Jan; Chalvatzaki, Georgia

doi:10.1109/humanoids53995.2022.10000239

Cited by 5 publications

(7 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2) Nuitrack Skeleton Interaction Dataset: (NuiSI) [6] is a dataset that we collected ourselves of the same 4 interactions as in [2]. The skeleton data of two human partners interacting with one another is recorded using two Intel Realsense D435 cameras (one recording each human partner).…”

Section: Experiments and Resultsmentioning

confidence: 99%

“…Learning a joint distribution over the degrees of freedom of a human and a robot has been widely used in learning HRI from demonstrations [5], [6], [27]- [31]. With a joint distribution, GMR provides a mathematically sound formulation of predicting the conditional distribution of the robot actions.…”

Section: A Gmr-based Interaction Dynamics With Mdnsmentioning

confidence: 99%

“…For the dataset from [2] and the NuiSI dataset [6], we use a VAE with 2 hidden layers each in the encoder and decoder with a dimensionality of (40,20) and (20,40) respectively, with Leaky ReLU activations and a 5D latent space. For the VAE posterior and the MDN outputs, we predict the mean and the log of the standard deviation (logstd).…”

Section: B Implementation Detailsmentioning

confidence: 99%

“…Vogt et al [7] showed the use of shared latent spaces in an Imitation Learning (IL) approach by decomposing interactions into multiple segments in a lower dimensional space with an underlying Gaussian Mixture Model (GMM) and learning the sequence of key poses using a Hidden Markov Model (HMM) to define the progression of an interaction. In our previous work, MILD [5], [6], we explore learning a shared latent space model using a Variational Autoencoder (VAE) wherein we learn a joint distribution over the trajectories of both the human and the robot using an HMM with underlying Gaussian States to represent the multimodality of the demonstrations. Rather than extracting key poses as in [7], we generate the robot's motion using Gaussian Mixture Regression (GMR) from the underlying HMM based on the human's observations in a reactive manner.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

MoVEInt: Mixture of Variational Experts for Learning Human–Robot Interactions From Demonstrations

Prasad,

Kshirsagar,

Koert

et al. 2024

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Shared dynamics models are important for capturing the complexity and variability inherent in Human-Robot Interaction (HRI). Therefore, learning such shared dynamics models can enhance coordination and adaptability to enable successful reactive interactions with a human partner. In this work, we propose a novel approach for learning a shared latent space representation for HRIs from demonstrations in a Mixture of Experts fashion for reactively generating robot actions from human observations. We train a Variational Autoencoder (VAE) to learn robot motions regularized using an informative latent space prior that captures the multimodality of the human observations via a Mixture Density Network (MDN). We show how our formulation derives from a Gaussian Mixture Regression formulation that is typically used approaches for learning HRI from demonstrations such as using an HMM/GMM for learning a joint distribution over the actions of the human and the robot. We further incorporate an additional regularization to prevent "mode collapse", a common phenomenon when using latent space mixture models with VAEs. We find that our approach of using an informative MDN prior from human observations for a VAE generates more accurate robot motions compared to previous HMM-based or recurrent approaches of learning shared latent representations, which we validate on various HRI datasets involving interactions such as handshakes, fistbumps, waving, and handovers. Further experiments in a real-world humanto-robot handover scenario show the efficacy of our approach for generating successful interactions with four different human interaction partners.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Section: A Gmr-based Interaction Dynamics With Mdnsmentioning

confidence: 99%

Section: B Implementation Detailsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

MoVEInt: Mixture of Variational Experts for Learning Human–Robot Interactions From Demonstrations

Prasad,

Kshirsagar,

Koert

et al. 2024

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Since many interactive tasks can naturally be broken down into underlying segments or phases that are then sequenced to achieve suitable behavior, previous works have explored learning HRI from demonstrations using Gaussian Mixture Models (GMMs) or, additionally, Hidden Markov Models (HMMs) with an underlying Mixture of Gaussians structure [3,4,13,14,11,10,15]. However, Hidden Markov Models have limitations in representing transition states.…”

Section: Introductionmentioning

confidence: 99%

Transition State Clustering for Interaction Segmentation and Learning

Hahne,

Prasad,

Kshirsagar

et al. 2024

Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction

Self Cite

View full text Add to dashboard Cite

Hidden Markov Models with an underlying Mixture of Gaussian structure have proven effective in learning Human-Robot Interactions from demonstrations for various interactive tasks via Gaussian Mixture Regression. However, a mismatch occurs when segmenting the interaction using only the observed state of the human compared to the joint state of the human and the robot. To enhance this underlying segmentation and subsequently the predictive abilities of such Gaussian Mixture-based approaches, we take a hierarchical approach by learning an additional mixture distribution on the states at the transition boundary. This helps prevent misclassifications that usually occur in such states. We find that our framework improves the performance of the underlying Gaussian Mixturebased approach, which we evaluate on various interactive tasks such as handshaking and fistbumps. CCS CONCEPTS• Theory of computation → Unsupervised learning and clustering; • Computing methodologies → Mixture modeling; Learning from demonstrations.

show abstract