Before, Between, and After: Enriching Robot Communication Surrounding Collaborative Creative Activities

Abstract

Research in creative robotics continues to expand across all creative domains, including art, music and language. Creative robots are primarily designed to be task specific, with limited research into the implications of their design outside their core task. In the case of a musical robot, this includes when a human sees and interacts with the robot before and after the performance, as well as in between pieces. These non-musical interaction tasks such as the presence of a robot during musical equipment set up, play a key role in the human perception of the robot however have received only limited attention. In this paper, we describe a new audio system using emotional musical prosody, designed to match the creative process of a musical robot for use before, between and after musical performances. Our generation system relies on the creation of a custom dataset for musical prosody. This system is designed foremost to operate in real time and allow rapid generation and dialogue exchange between human and robot. For this reason, the system combines symbolic deep learning through a Conditional Convolution Variational Auto-encoder, with an emotion-tagged audio sampler. We then compare this to a SOTA text-to-speech system in our robotic platform, Shimon the marimba player.We conducted a between-groups study with 100 participants watching a musician interact for 30 s with Shimon. We were able to increase user ratings for the key creativity metrics; novelty and coherence, while maintaining ratings for expressivity across each implementation. Our results also indicated that by communicating in a form that relates to the robot’s core functionality, we can raise likeability and perceived intelligence, while not altering animacy or anthropomorphism. These findings indicate the variation that can occur in the perception of a robot based on interactions surrounding a performance, such as initial meetings and spaces between pieces, in addition to the core creative algorithms.

Social robotics in music education: A systematic review

“…Finally, we would like to highlight that the studies under examination provided evidence of positive effects on students' learning in the educational context related to music. This is also noted in previous studies (Cádiz et al, 2021;Chakraborty et al, 2021;Savery et al, 2021). Nonetheless, these results must be interpreted very cautiously, due to the limitations to generalize them because the sample used is so small.…”

“…By way of example, Savery et al (2021) analyzed the musician's interaction with the robot before and after the performance, as well as between pieces. According to these scholars, these tasks involving a non-musical interaction, such as the presence of a robot during the configuration of the musical team, play a key role in the robot's human perception.…”

Robotic Arm Generative Painting Through Real-time Analysis of Music Performance

“…There have also been multiple projects extending findings from robotic musicianship into other fields. These include using approaches from robotic musicianship for real-time music generation to interact with individuals to complete basic tasks [17,18]. Other applications include in education for children with autism [19] and custom devices for interaction [20].…”