Synthesizing Skeletal Motion and Physiological Signals as a Function of a Virtual Human's Actions and Emotions

Banerjee, Bonny; Kapourchali, Masoumeh Heidari; Baruah, Murchana; Deb, Mousumi; Sakauye, Kenneth; Olufsen, Mette S.

doi:10.1137/1.9781611976700.77

Cited by 4 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such data are scarce, primarily due to restrictions on usage of soft and hard sensors to collect data due to risk of confidentiality breach and privacy invasion. An alternative is to generate data using a combination of physics-based and generative AI models (see [ 61 ], for example).…”

Section: Discussionmentioning

confidence: 99%

Attention-Based Variational Autoencoder Models for Human–Human Interaction Recognition via Generation

Banerjee,

Baruah

2024

Sensors

View full text Add to dashboard Cite

The remarkable human ability to predict others’ intent during physical interactions develops at a very early age and is crucial for development. Intent prediction, defined as the simultaneous recognition and generation of human–human interactions, has many applications such as in assistive robotics, human–robot interaction, video and robotic surveillance, and autonomous driving. However, models for solving the problem are scarce. This paper proposes two attention-based agent models to predict the intent of interacting 3D skeletons by sampling them via a sequence of glimpses. The novelty of these agent models is that they are inherently multimodal, consisting of perceptual and proprioceptive pathways. The action (attention) is driven by the agent’s generation error, and not by reinforcement. At each sampling instant, the agent completes the partially observed skeletal motion and infers the interaction class. It learns where and what to sample by minimizing the generation and classification errors. Extensive evaluation of our models is carried out on benchmark datasets and in comparison to a state-of-the-art model for intent prediction, which reveals that classification and generation accuracies of one of the proposed models are comparable to those of the state of the art even though our model contains fewer trainable parameters. The insights gained from our model designs can inform the development of efficient agents, the future of artificial intelligence (AI).

show abstract

Section: Discussionmentioning

confidence: 99%

Attention-Based Variational Autoencoder Models for Human–Human Interaction Recognition via Generation

Banerjee,

Baruah

2024

Sensors

View full text Add to dashboard Cite

show abstract

“…This will allow generation of potentially infinite amounts of shareable data. Also, simulated humans will allow evaluation of artificial intelligence (AI) agents 1 , implemented in software, in terms of fidelity to human physiological performance. Virtual humans are ubiquitous.…”

Section: Introductionmentioning

confidence: 99%

“…However, a framework to derive physical and physiological signals from a virtual human round-the-clock as a function of his day-to-day actions and emotions is currently missing. 1 An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators [2]. Such agents, implemented in software, have been reported in our prior work [3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] as well as in others'.…”

Section: Introductionmentioning

confidence: 99%

Synthesizing Skeletal Motion and Physiological Signals as a Function of a Virtual Human's Actions and Emotions

Banerjee¹,

Kapourchali²,

Baruah³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Round-the-clock monitoring of human behavior and emotions is required in many healthcare applications which is very expensive but can be automated using machine learning (ML) and sensor technologies. Unfortunately, the lack of infrastructure for collection and sharing of such data is a bottleneck for ML research applied to healthcare. Our goal is to circumvent this bottleneck by simulating a human body in virtual environment. This will allow generation of potentially infinite amounts of shareable data from an individual as a function of his actions, interactions and emotions in a care facility or at home, with no risk of confidentiality breach or privacy invasion. In this paper, we develop for the first time a system consisting of computational models for synchronously synthesizing skeletal motion, electrocardiogram, blood pressure, respiration, and skin conductance signals as a function of an open-ended set of actions and emotions. Our experimental evaluations, involving user studies, benchmark datasets and comparison to findings in the literature, show that our models can generate skeletal motion and physiological signals with high fidelity. The proposed framework is modular and allows the flexibility to experiment with different models. In addition to facilitating ML research for round-the-clock monitoring at a reduced cost, the proposed framework will allow reusability of code and data, and may be used as a training tool for ML practitioners and healthcare professionals.

show abstract

A review of platforms for simulating embodied agents in 3D virtual environments

2022

View full text Add to dashboard Cite

Synthesizing Skeletal Motion and Physiological Signals as a Function of a Virtual Human's Actions and Emotions

Cited by 4 publications

References 0 publications

Attention-Based Variational Autoencoder Models for Human–Human Interaction Recognition via Generation

Attention-Based Variational Autoencoder Models for Human–Human Interaction Recognition via Generation

Synthesizing Skeletal Motion and Physiological Signals as a Function of a Virtual Human's Actions and Emotions

A review of platforms for simulating embodied agents in 3D virtual environments

Contact Info

Product

Resources

About