Development of Performance System With Musical Dynamics Expression on Humanoid Saxophonist Robot

Lin, Jie; Kawai, Mao; Nishio, Yuya; Cosentino, Sarah; Takanishi, Atsuo

doi:10.1109/lra.2019.2897372

Cited by 16 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a pattern recognition task, there are plenty of classification methods that can be adopted to classify facial emotions [11]- [13], [20]- [28]. In [11], [12], three-and twolayer fuzzy support vector regression-Takagi-Sugeno models are suggested for the emotion understanding in human-robot-interaction (HRI) task, e.g., the drink reflecting different emotions, human following [5]. Their average video-based recognition rates for different genders, provinces, and ages are ordinary.…”

Section: Related Workmentioning

confidence: 99%

“…In [3], the suggested robot Mortimer including social behaviors can increase engagement and social presence; in addition, the effect of extending weekly collocated musical improvisation sessions is investigated by making Mortimer an active member of the participant's virtual social network. In [5], specific human following through machine learning of SSD-FN-KCF is developed. In [6], musical robots are designed to control those dynamics, articulation, and tempo to give the audience an experience as compared with listening to a professional human musician.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Human–Robot Collaboration Using Sequential-Recurrent-Convolution-Network-Based Dynamic Face Emotion and Wireless Speech Command Recognitions

Hwang

Deng

2023

IEEE Access

View full text Add to dashboard Cite

The proposed sequential recurrent convolution network (SRCN) includes two parts: one convolution neural network (CNN) and a sequence of long short-term memory (LSTM) models. The CNN is to achieve the feature vector of face emotion or speech command. Then, a sequence of LSTM models with the shared weight reflects a sequence of inputs provided by a (pre-trained) CNN with a sequence of input sub-images or spectrograms corresponding to face emotion and speech command, respectively. Simply put, one SRCN for dynamic face emotion recognition (SRCN-DFER) and another SRCN for wireless speech command recognition (SRCN-WSCR) are developed. The proposed approach not only effectively tackles the recognitions of dynamic mapping of face emotion and speech command with average generalized recognition rate of 98% and 96.7% but also prevents the overfitting problem in a noisy environment. The comparisons among mono and stereo visions, Deep CNN, and ResNet50 confirm the superiority of the proposed SRCN-DFER. The comparisons among SRCN-WSCR with noise-free data, SRCN-WSCR with noisy data, and multiclass support vector machine validate its robustness. Finally, the human-robot collaboration (HRC) using our developed omnidirectional service robot, including human and face detections, trajectory tracking by the previously designed adaptive stratified finite-time saturated control, face emotion and speech command recognitions, and music play, validates the effectiveness, feasibility, and robustness of the proposed method.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Human–Robot Collaboration Using Sequential-Recurrent-Convolution-Network-Based Dynamic Face Emotion and Wireless Speech Command Recognitions

Hwang

Deng

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…We created a new flute-playing robot to collect recordings of its performance for use in learning. Two types of robots exist that can play musical instruments: those that aim to generate performances without reproducing human physical characteristics [13][14][15], and those that aim to achieve a more human-like performance by reproducing human body parts more precisely [16,17]. The latter type of robot can reproduce tones and playing techniques closer to those of humans, but it is more costly to develop.…”

Section: Flute-playing Robotmentioning

confidence: 99%

“…The mouth of a human has complex parameters such as the shape and viscosity of the oral cavity and the shape of the lips. The accuracy of parameter estimation can be further improved by creating a sophisticated artificial mouth that takes these features into account, such as in [16].…”

Section: (B) Parameter Expansionmentioning

confidence: 99%

Sensing Control Parameters of Flute from Microphone Sound Based on Machine Learning from Robotic Performer

Kuroda

Koutaki

2022

Sensors

View full text Add to dashboard Cite

When learning to play a musical instrument, it is important to improve the quality of self-practice. Many systems have been developed to assist practice. Some practice assistance systems use special sensors (pressure, flow, and motion sensors) to acquire the control parameters of the musical instrument, and provide specific guidance. However, it is difficult to acquire the control parameters of wind instruments (e.g., saxophone or flute) such as flow and angle between the player and the musical instrument, since it is not possible to place sensors into the mouth. In this paper, we propose a sensorless control parameter estimation system based on the recorded sound of a wind instrument using only machine learning. In the machine learning framework, many training samples that have both sound and correct labels are required. Therefore, we generated training samples using a robotic performer. This has two advantages: (1) it is easy to obtain many training samples with exhaustive control parameters, and (2) we can use the correct labels as the given control parameters of the robot. In addition to the samples generated by the robot, some human performance data were also used for training to construct an estimation model that enhanced the feature differences between robot and human performance. Finally, a flute control parameter estimation system was developed, and its estimation accuracy for eight novice flute players was evaluated using the Spearman’s rank correlation coefficient. The experimental results showed that the proposed system was able to estimate human control parameters with high accuracy.

show abstract

“…There are numerous applications using autonomous robot for instance engineering education, 1 food service robot, 2 delivery robot, 3 patrol robot, 4 assistive robot, 5 or entertain robot. 6 Most of these working environments are crowded and complex area which is difficult for the autonomous system. Once, a mistake in controlling, programming or operating occurs, the extreme casualty would effect on human.…”

Section: Introductionmentioning

confidence: 99%

Develop the socially human-aware navigation system using dynamic window approach and optimize cost function for autonomous medical robot

Ngo

Thien

et al. 2014

Advances in Mechanical Engineering

View full text Add to dashboard Cite

In previous works, the perceived safety and comfort are currently not the principal objectives of all industries, especially robotics system. It might lead not to take psychological safety into consideration of adjusting robot behavior, hence, the human-robot interaction lacks of ease and naturalness. In this paper, a novel framework of human’s zones to ensure safety for social interactions in human-machine system is proposed. In the context of service robot in hospital, machine should not produce any actions that may induce worry, surprise or bother. To maintain the comfortable interaction, an algorithm to update human’s state into personal space is developed. Then, a motion model of robot is demonstrated with assumption of the reference path under segmentation. Dynamic Window Approach is employed for motion planning while Optimize Cost function searches the shortest path in a graph. To validate our approach, three test cases (without human-aware framework, with basic model of human’s zone and with extended personal space) are carried out in the same context. Moreover, three interactive indicators, for instance collision index (CI), interaction index (CII) and relative velocity of robot (Vr), are analyzed in different situations. Lack of human-aware framework, robot might break all thresholds and meet the potential collisions. While robot with basic model of human’s zone in its perception maintains the physically safe thresholds but not socially, it respects whole criterions in both physical constraints and social relations. As a result, our findings are useful for robot’s navigation in presence of human while the socially comfortable interaction is guaranteed.

show abstract

Development of Performance System With Musical Dynamics Expression on Humanoid Saxophonist Robot

Cited by 16 publications

References 18 publications

Human–Robot Collaboration Using Sequential-Recurrent-Convolution-Network-Based Dynamic Face Emotion and Wireless Speech Command Recognitions

Human–Robot Collaboration Using Sequential-Recurrent-Convolution-Network-Based Dynamic Face Emotion and Wireless Speech Command Recognitions

Sensing Control Parameters of Flute from Microphone Sound Based on Machine Learning from Robotic Performer

Develop the socially human-aware navigation system using dynamic window approach and optimize cost function for autonomous medical robot

Contact Info

Product

Resources

About