SIGVerse: A Cloud-Based VR Platform for Research on Multimodal Human-Robot Interaction

Inamura, Tetsunari; Mizuchi, Yoshiaki

doi:10.3389/frobt.2021.549360

Cited by 44 publications

(17 citation statements)

References 66 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We show that the proposed method can improve the efficiency of path planning. The simulator environment was SIGVerse, version 3.0 [25]. The virtual robot model in SIGVerse is the Toyota Human Support Robot (HSR).…”

Section: Experiments I: Planning Tasks In Simulatormentioning

confidence: 99%

Spatial Concept-based Topometric Semantic Mapping for Hierarchical Path-planning from Speech Instructions

Taniguchi¹,

Shuya²,

Taniguchi³

2022

Preprint

View full text Add to dashboard Cite

Navigating to destinations using human speech instructions is an important task for autonomous mobile robots that operate in the real world. Spatial representations include a semantic level that represents an abstracted location category, a topological level that represents their connectivity, and a metric level that depends on the structure of the environment. The purpose of this study is to realize a hierarchical spatial representation using a topometric semantic map and planning efficient paths through human-robot interactions. We propose a novel probabilistic generative model, SpCoTMHP, that forms a topometric semantic map that adapts to the environment and leads to hierarchical path planning. We also developed approximate inference methods for path planning, where the levels of the hierarchy can influence each other. The proposed path planning method is theoretically supported by deriving a formulation based on control as probabilistic inference. The navigation experiment using human speech instruction shows that the proposed spatial concept-based hierarchical path planning improves the performance and reduces the calculation cost compared with conventional methods. Hierarchical spatial representation provides a mutually understandable form for humans and robots to render language-based navigation tasks feasible.

show abstract

Section: Experiments I: Planning Tasks In Simulatormentioning

confidence: 99%

Spatial Concept-based Topometric Semantic Mapping for Hierarchical Path-planning from Speech Instructions

Taniguchi¹,

Shuya²,

Taniguchi³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A study on the spatial concept acquisition task for robots (Taniguchi et al, 2020a), SIGVerse, (Inamura and Mizuchi, 2021) used an architecture that connects Unity and ROS. As in their study, we used a virtual home environment 2 in Unity.…”

Section: Conditionsmentioning

confidence: 99%

Automatic selection of coordinate systems for learning relative and absolute spatial concepts

et al. 2022

View full text Add to dashboard Cite

Robots employed in homes and offices need to adaptively learn spatial concepts using user utterances. To learn and represent spatial concepts, the robot must estimate the coordinate system used by humans. For example, to represent spatial concept “left,” which is one of the relative spatial concepts (defined as a spatial concept depending on the object’s location), humans use a coordinate system based on the direction of a reference object. As another example, to represent spatial concept “living room,” which is one of the absolute spatial concepts (defined as a spatial concept that does not depend on the object’s location), humans use a coordinate system where a point on a map constitutes the origin. Because humans use these concepts in daily life, it is important for the robot to understand the spatial concepts in different coordinate systems. However, it is difficult for robots to learn these spatial concepts because humans do not clarify the coordinate system. Therefore, we propose a method (RASCAM) that enables a robot to simultaneously estimate the coordinate system and spatial concept. The proposed method is based on ReSCAM+O, which is a learning method for relative spatial concepts based on a probabilistic model. The proposed method introduces a latent variable that represents a coordinate system for simultaneous learning. This method can simultaneously estimate three types of unspecified information: coordinate systems, reference objects, and the relationship between concepts and words. No other method can estimate all these three types. Experiments using three different coordinate systems demonstrate that the proposed method can learn both relative and absolute spatial concepts while accurately selecting the coordinate system. The proposed approach can be beneficial for service robots to flexibly understand a new environment through the interactions with humans.

show abstract

“…Not only that, multimodal robots are being used more and more in the performing arts. Robot performing art refers to the combination of robot technology and performing art to create a unique and amazing art form (Inamura and Mizuchi, 2021 ). It uses robots as the main body or participants of the performance, and through the robot's movements, music, lighting, images and other elements, it presents the audience with a visual feast that integrates technology and art.…”

Section: Introductionmentioning

confidence: 99%

Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception

Wang

2024

Front. Neurorobot.

View full text Add to dashboard Cite

The field of multimodal robotic musical performing arts has garnered significant interest due to its innovative potential. Conventional robots face limitations in understanding emotions and artistic expression in musical performances. Therefore, this paper explores the application of multimodal robots that integrate visual and auditory perception to enhance the quality and artistic expression in music performance. Our approach involves integrating GRU (Gated Recurrent Unit) and GoogLeNet models for sentiment analysis. The GRU model processes audio data and captures the temporal dynamics of musical elements, including long-term dependencies, to extract emotional information. The GoogLeNet model excels in image processing, extracting complex visual details and aesthetic features. This synergy deepens the understanding of musical and visual elements, aiming to produce more emotionally resonant and interactive robot performances. Experimental results demonstrate the effectiveness of our approach, showing significant improvements in music performance by multimodal robots. These robots, equipped with our method, deliver high-quality, artistic performances that effectively evoke emotional engagement from the audience. Multimodal robots that merge audio-visual perception in music performance enrich the art form and offer diverse human-machine interactions. This research demonstrates the potential of multimodal robots in music performance, promoting the integration of technology and art. It opens new realms in performing arts and human-robot interactions, offering a unique and innovative experience. Our findings provide valuable insights for the development of multimodal robots in the performing arts sector.

show abstract

SIGVerse: A Cloud-Based VR Platform for Research on Multimodal Human-Robot Interaction

Cited by 44 publications

References 66 publications

Spatial Concept-based Topometric Semantic Mapping for Hierarchical Path-planning from Speech Instructions

Spatial Concept-based Topometric Semantic Mapping for Hierarchical Path-planning from Speech Instructions

Automatic selection of coordinate systems for learning relative and absolute spatial concepts

Multimodal robotic music performance art based on GRU-GoogLeNet model fusing audiovisual perception

Contact Info

Product

Resources

About