Style Transfer for Co-speech Gesture Animation: A Multi-speaker Conditional-Mixture Approach

Ahuja, Chaitanya; Lee, Jun Young; Nakano, Yukiko I.; Morency, Louis–Philippe

doi:10.1007/978-3-030-58523-5_15

Cited by 62 publications

(74 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…An adversarial discriminator is used to evaluate if the generated motions match the style of the speaker. The method for generating co-speech gestures proposed by Ahuja et al [200] in 2020 also aimed at reflecting the gestural style of different speakers, while maintaining the content of the gesture. The style of the speaker is encoded in a 2D space, and defined by the idiosyncrasy of each speaker and a series of contextual circumstances, like the orientation of the body, or the posture (standing versus sitting, for example).…”

Section: Comparison Of Co-speech Gesture Prediction/generation Methodsmentioning

confidence: 99%

“…Other approaches include the use of prosodic features, as shown by the work of Chiu et al [190], or directly encoding the audio signal. This last method appears in the approaches presented by Yu et al [206] or Ahuja et al [200]. Some works combine both audio features and text to improve the obtained results.…”

Section: Multimodalitymentioning

confidence: 99%

“…Besides the use of text, audio, or both to represent the content of the speech, multiple authors decided to constrain the gestures generated by their models according to different external factors. For example, both Ahuja et al [200] and Ginosar et al [199] presented methods for generating behaviours that represent the style of the speaker. While the model proposed by Ginosar et al can only be trained to convey the style of a particular speaker, Ahuja et al used a multi-generator structure that allows the system to receive the desired style as an input, and use it during the generation process.…”

Section: Reference Algorithm Usedmentioning

confidence: 99%

See 2 more Smart Citations

A Social Robot Assisting in Cognitive Stimulation Therapy

Salichs

Fernández-Rodicio

Castillo

et al. 2018

Advances in Practical Applications of Agents, Multi-Agent Systems, and Complexity: The PAAMS Collection

View full text Add to dashboard Cite

Society is experiencing a series of demographic changes that can result in an unbalance between the active working and non-working age populations. One of the solutions considered to mitigate this problem is the inclusion of robots in multiple sectors, including the service sector. But for this to be a viable solution, among other features, robots need to be able to interact with humans successfully. This thesis seeks to endow a social robot with the abilities required for a natural human-robot interactions. The main objective is to contribute to the body of knowledge on the area of Human-Robot Interaction with a new, platform-independent, modular approach that focuses on giving roboticists the tools required to develop applications that involve interactions with humans. In particular, this thesis focuses on three problems that need to be addressed: (i) modelling interactions between a robot and an user; (ii) endow the robot with the expressive capabilities required for a successful communication; and (iii) endow the robot with a lively appearance.The approach to dialogue modelling presented in this thesis proposes to model dialogues as a sequence of atomic interaction units, called Communicative Acts, or CAs. They can be parametrized in runtime to achieve different communicative goals, and are endowed with mechanisms oriented to solve some of the uncertainties related to interaction. Two dimensions have been used to identify the required CAs: initiative (the robot or the user), and intention (either retrieve information or to convey it). These basic CAs can be combined in a hierarchical manner to create more re-usable complex structures. This approach simplifies the creation of new interactions, by allowing developers to focus exclusively on designing the flow of the dialogue, without having to re-implement functionalities that are common to all dialogues (like error handling, for example).The expressiveness of the robot is based on the use of a library of predefined multimodal gestures, or expressions, modelled as state machines. The module managing the expressiveness receives requests for performing gestures, schedules their execution in order to avoid any possible conflict that might arise, loads them, and ensures that their execution goes without problems. The proposed approach is also able to generate expressions in runtime based on a list of unimodal actions (an utterance, the motion of a limb, etc...). One of the key features of the proposed expressiveness management approach is the integration of a series of modulation techniques that can be used to modify the robot's expressions in runtime. This would allow the robot to adapt them to the particularities of a given situation (which would also increase the variability of the robot expressiveness), and to display different internal states with the same expressions.

show abstract

Section: Comparison Of Co-speech Gesture Prediction/generation Methodsmentioning

confidence: 99%

Section: Multimodalitymentioning

confidence: 99%

Section: Reference Algorithm Usedmentioning

confidence: 99%

See 1 more Smart Citation

A Social Robot Assisting in Cognitive Stimulation Therapy

Salichs

Fernández-Rodicio

Castillo

et al. 2018

Advances in Practical Applications of Agents, Multi-Agent Systems, and Complexity: The PAAMS Collection

View full text Add to dashboard Cite

show abstract

“…Style control vectors were inputted to the neural network model, and it was demonstrated that the model generates gestures following the inputted style intents. Also, speaker identities were used to generate stylized gestures reflecting inter-person variability [1,44]. A style embedding space was trained from a large corpus of gestures of different speakers in [44].…”

Section: Controllable Gesture Generationmentioning

confidence: 99%

“…A style embedding space was trained from a large corpus of gestures of different speakers in [44]. A style disentanglement was studied in [1].…”

Section: Controllable Gesture Generationmentioning

confidence: 99%

SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents

Yoon

Park

Jang

et al. 2021

The 34th Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

Non-verbal behavior is essential for embodied agents like social robots, virtual avatars, and digital humans. Existing behavior authoring approaches including keyframe animation and motion capture are too expensive to use when there are numerous utterances requiring gestures. Automatic generation methods show promising results, but their output quality is not satisfactory yet, and it is hard to modify outputs as a gesture designer wants. We introduce a new gesture generation toolkit, named SGToolkit, which gives a higher quality output than automatic methods and is more efficient than manual authoring. For the toolkit, we propose a neural generative model that synthesizes gestures from speech and accommodates fine-level pose controls and coarse-level style controls from users. The user study with 24 participants showed that the toolkit is favorable over manual authoring, and the generated gestures were also human-like and appropriate to input speech. The SGToolkit is platform agnostic, and the code is available at https://github.com/ai4r/SGToolkit. CCS Concepts: • Human-centered computing → Interaction techniques; Interactive systems and tools; • Computer systems organization → Robotics.

show abstract

Audio-Driven Stylized Gesture Generation with Flow-Based Model

Wen

Sun

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Style Transfer for Co-speech Gesture Animation: A Multi-speaker Conditional-Mixture Approach

Cited by 62 publications

References 38 publications

A Social Robot Assisting in Cognitive Stimulation Therapy

A Social Robot Assisting in Cognitive Stimulation Therapy

SGToolkit: An Interactive Gesture Authoring Toolkit for Embodied Conversational Agents

Audio-Driven Stylized Gesture Generation with Flow-Based Model

Contact Info

Product

Resources

About