Automatic text‐to‐gesture rule generation for embodied conversational agents

Ghazanfar, Ali; Lee, Myungho; Hwang, Jae-In

doi:10.1002/cav.1944

Cited by 15 publications

(16 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We demonstrate realistic interactions between a virtual human and real animal dolls using the silhouette mesh. Our virtual human is equipped with voice chat [3], gesture animation [47] to provide interactive scenarios (see Figure 7). While playing with the dolls, the virtual human will point or approach a doll to draw the user's attention, and occasionally exhibit the behavior of touching the doll plausibly to make the experience more appealing.…”

Section: Case Study: Users Play With Animal Dolls In Mobile Ar Applicationmentioning

confidence: 99%

Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality

et al. 2021

Self Cite

View full text Add to dashboard Cite

Realistic interactions with real objects (e.g., animals, toys, robots) in an augmented reality (AR) environment enhances the user experience. The common AR apps on the market achieve realistic interactions by superimposing pre-modeled virtual proxies on the real objects in the AR environment. This way user perceives the interaction with virtual proxies as interaction with real objects. However, catering to environment change, shape deformation, and view update is not a trivial task. Our proposed method uses the dynamic silhouette of a real object to enable realistic interactions. Our approach is practical, lightweight, and requires no additional hardware besides the device camera. For a case study, we designed a mobile AR application to interact with real animal dolls. Our scenario included a virtual human performing four types of realistic interactions. Results demonstrated our method’s stability that does not require pre-modeled virtual proxies in case of shape deformation and view update. We also conducted a pilot study using our approach and reported significant improvements in user perception of spatial awareness and presence for realistic interactions with a virtual human.

show abstract

Section: Case Study: Users Play With Animal Dolls In Mobile Ar Applicationmentioning

confidence: 99%

Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…For example, SimSensei Kiosk [4] demonstrated VA's listening gestures, such as nodding and gazing, during the clinical interview to build rapport with the client. Ali et al [5] focused on improving the naturalness of conversational interactions with hand gestures (which unconsciously express engagement while speaking) and synchronizing it with speech. Another study explored the impact of VA's body size on user experience in augmented reality (AR) [15], [16].…”

Section: Virtual Agents With Engaging Behaviormentioning

confidence: 99%

“…Engaging behavior (E) aims to make the user aware of the VA's active engagement in the group discussion. Our study employed four non-verbal behaviors, which also have been commonly used in prior studies [4], [5], [7] -namely, mutual gaze, directed posture, listening gestures, and speaking gestures (also listed in Table 1). In the group discussion, the VA has two roles: to listen to the other participants (as a listener) and speak to them (as a speaker).…”

Section: Engaging and Non-engaging Behavior (E/ne)mentioning

confidence: 99%

“…When speaking, the VA provides engaging behaviors by using co-speech gestures to express active participation (see Figure 1 (a) and (b)). To generate semantically relevant gestures (mostly arms and hands movement), we employed a deep learning model [5] trained with motion capture videos and transcripts. These gestures initiate when it begins speaking and is displayed toward the participant by the listener detection method (see Section 3.4.2).…”

Section: Engaging and Non-engaging Behavior (E/ne)mentioning

confidence: 99%

“…Through their voice and facial expressions, the VAs can evoke a range of emotions or affectivness [4]. The VA's engagements can also be perceived through their eye gaze and body gestures [4], [5]. It has been demonstrated that these engagement and affectivness are key components in the modulation of how human interlocutors perceive and interact with the VAs [6], [7].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Impact of Visual Art and High Affective Arousal on Heuristic Decision-Making in Consumers

et al. 2020

View full text Add to dashboard Cite

In marketing, the use of visual-art-based designs on products or packaging crucially impacts consumers’ decision-making when purchasing. While visual art in product packaging should be designed to induce consumer’s favorable evaluations, it should not evoke excessive affective arousal, because this may lead to the depletion of consumer’s cognitive resources. Thus, consumers may use heuristic decision-making and commit an inadvertent mistake while purchasing. Most existing studies on visual arts in marketing have focused on preference (i.e., affective valence) using subjective evaluations. To address this, we applied a neuroscientific measure, electroencephalogram (EEG) to increase experimental validity. Two successive tasks were designed to examine the effects of affective arousal and affective valence, evoked by visual artwork, on the consecutive cognitive decision-making. In task 1, to evaluate the effect of visual art, EEG of two independent groups of people was measured when they viewed abstract artwork. The abstract art of neoplasticism (AbNP) group (n = 20) was showing Mondrian’s artwork, while the abstract art of expressionism (AbEX) group (n = 18) viewed Kandinsky’s artwork. The neoplasticism movement strove to eliminate emotion in art and expressionism to express the feelings of the artist. Building on Gallese’s embodied simulation theory, AbNP and AbEX artworks were expected to induce lower and higher affect, respectively. In task 2, we investigated how the induced affect differentially influenced a succeeding cognitive Stroop task. We anticipated that the AbEX group would deplete more cognitive resources than AbNP group, based on capacity limitation theory. Significantly stronger affect was induced in the AbEX group in task 1 than in the AbNP group, especially in affective arousal. In task 2, the AbEX group showed a faster reaction time and higher error rate in the Stroop task. According to our hypotheses, the higher affective arousal state of the AbEX group might deplete more cognitive resources during task 1 and result in poorer performance in task 2 because affect impacted their cognitive resources. This is the first study using neuroscientific measures to prove that high affective arousal induced by visual arts on packaging may induce heuristic decision-making in consumers, thereby advancing our understanding of neuromarketing.

show abstract

Enhancing doctor‐patient communication in surgical explanations: Designing effective facial expressions and gestures for animated physician characters

Kim,

Ali,

Hwang

2024

Computer Animation & Virtual

Self Cite

View full text Add to dashboard Cite

Paying close attention to facial expressions, gestures, and communication techniques is essential when creating animated physician characters that are realistic and captivating when describing surgical procedures. This paper emphasizes the integration of appropriate emotions, co‐speech gestures when medical experts explain the medical procedure, and designing animated characters. We can achieve healthy doctor‐patient relationships and improvement of patients' understanding by depicting these components truthfully. We suggest two critical approaches to developing virtual medical experts by incorporating these elements. First, doctors can generate the contents of the surgical procedure with a virtual doctor. Second, patients can listen to the surgical procedure described by the virtual doctor and ask if they have any questions. Our system helps patients by considering their psychology and adding medical professionals' opinions. These improvements ensure the animated virtual agent is comforting, reassuring, and emotionally supportive. Through a user study, we evaluated our hypothesis and gained insight into improvements.

show abstract

Automatic text‐to‐gesture rule generation for embodied conversational agents

Cited by 15 publications

References 24 publications

Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality

Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality

The Impact of Visual Art and High Affective Arousal on Heuristic Decision-Making in Consumers

Enhancing doctor‐patient communication in surgical explanations: Designing effective facial expressions and gestures for animated physician characters

Contact Info

Product

Resources

About