Optimization of criterion for objective evaluation of HRI performance that approximates subjective evaluation: a case study in robot competition

Mizuchi, Yoshiaki; Inamura, Tetsunari

doi:10.1080/01691864.2019.1698462

Cited by 13 publications

(5 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results obtained indicate that the subjective evaluation result was significantly influenced by the distance traveled by the test subjects and the changes in the gaze directions of the test subjects (Mizuchi and Inamura, 2020). The coefficient for the explanatory variable any_inst has a large value, which is a variable indicating whether or not the robot generated instructions, corresponding to the software bug that interrupted the robot behavior.…”

Section: Modeling Of Subjective Evaluation Of Hri Qualitymentioning

confidence: 95%

“…Although the best evaluation method involves asking several referees to score the performance in several trials, human navigation was evaluated by a certain regulation, such as a positive point for "desirable behavior" and a negative point for "unfriendly behavior," which is described in the rulebook from a subjective viewpoint. We have addressed this challenge to determine the dominant factor for the evaluation of the interaction behavior in the HRI dataset (Mizuchi and Inamura, 2020). The approach is to have a third party evaluate the quality of human-robot interaction, and model the relationship between the subjective evaluation points and the physical and social behaviors of humans and robots.…”

Section: Modeling Of Subjective Evaluation Of Hri Qualitymentioning

confidence: 99%

“…Regression coefficients obtained for each objective factors(Mizuchi and Inamura, 2020). Time taken to complete the task β time task comp −0.0145 a Time_grasp: Time taken to complete grasping of the target object β time grasp −0.0022 a Num_incorr_grasp: Number of incorrect object-grasps β num incorr grasp −0.0017 Any_inst: Whether the robot was able to generate any instructions β…”

mentioning

confidence: 99%

See 2 more Smart Citations

SIGVerse: A Cloud-Based VR Platform for Research on Multimodal Human-Robot Interaction

Inamura

Mizuchi

2021

Front. Robot. AI

View full text Add to dashboard Cite

Research on Human-Robot Interaction (HRI) requires the substantial consideration of an experimental design, as well as a significant amount of time to practice the subject experiment. Recent technology in virtual reality (VR) can potentially address these time and effort challenges. The significant advantages of VR systems for HRI are: 1) cost reduction, as experimental facilities are not required in a real environment; 2) provision of the same environmental and embodied interaction conditions to test subjects; 3) visualization of arbitrary information and situations that cannot occur in reality, such as playback of past experiences, and 4) ease of access to an immersive and natural interface for robot/avatar teleoperations. Although VR tools with their features have been applied and developed in previous HRI research, all-encompassing tools or frameworks remain unavailable. In particular, the benefits of integration with cloud computing have not been comprehensively considered. Hence, the purpose of this study is to propose a research platform that can comprehensively provide the elements required for HRI research by integrating VR and cloud technologies. To realize a flexible and reusable system, we developed a real-time bridging mechanism between the robot operating system (ROS) and Unity. To confirm the feasibility of the system in a practical HRI scenario, we applied the proposed system to three case studies, including a robot competition named RoboCup@Home. via these case studies, we validated the system’s usefulness and its potential for the development and evaluation of social intelligence via multimodal HRI.

show abstract

Section: Modeling Of Subjective Evaluation Of Hri Qualitymentioning

confidence: 95%

Section: Modeling Of Subjective Evaluation Of Hri Qualitymentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

SIGVerse: A Cloud-Based VR Platform for Research on Multimodal Human-Robot Interaction

Inamura

Mizuchi

2021

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…Hence, we first investigate (RQ1:) whether pre-training robot behaviours with crowd-sourced data improves robo-waiter perceptions as compared to a 'control' condition with randomly sampled, static robot behaviours. Recent works have also highlighted the importance of using objective feedback in HRI evaluation, arguing that the timeconsuming nature of subjective feedback (via questionnaires) makes it less likely to be effective in longitudinal learning settings such as with HSRs [43]. Thus, we employ an objective evaluation strategy, via explicit (using speech) and implicit (using facial expressions) feedback given by the participants during their interactions with the HSR.…”

Section: Research Questions and Contributionsmentioning

confidence: 99%

Learning Socially Appropriate Robo-waiter Behaviours through Real-time User Feedback

McQuillin

Churamani

Güneş

2022

2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)

View full text Add to dashboard Cite

Current Humanoid Service Robot (HSR) behaviours mainly rely on static models that cannot adapt dynamically to meet individual customer attitudes and preferences. In this work, we focus on empowering HSRs with adaptive feedback mechanisms driven by either implicit reward, by estimating facial affect, or explicit reward, by incorporating verbal responses of the human 'customer'. To achieve this, we first create a custom dataset, annotated using crowd-sourced labels, to learn appropriate approach (positioning and movement) behaviours for a Robowaiter. This dataset is used to pre-train a Reinforcement Learning (RL) agent to learn behaviours deemed socially appropriate for the robo-waiter. This model is later extended to include separate implicit and explicit reward mechanisms to allow for interactive learning and adaptation from user social feedback. We present a within-subjects Human-Robot Interaction (HRI) study with 21 participants implementing interactions between the robowaiter and human customers implementing the above-mentioned model variations. Our results show that both explicit and implicit adaptation mechanisms enabled the adaptive robo-waiter to be rated as more enjoyable and sociable, and its positioning relative to the participants as more appropriate compared to using the pre-trained model or a randomised control implementation.

show abstract

“…Accordingly, if the robot's instructions are insufficient/inaccurate, the subject wanders around and makes mistakes in the task. The subjects' response is evaluated to determine if the robot's guidance is accurate, efficient, friendly, and easily understood by the subjects [6].…”

Section: Hri In a Robot Competitionmentioning

confidence: 99%

VR platform enabling crowdsourcing of embodied HRI experiments – case study of online robot competition

2021

View full text Add to dashboard Cite

The COVID-19 pandemic has made it difficult to conduct research involving human-robot interactions (HRIs). As a result, many robot competitions have been canceled in the past year. To address this problem, we propose a cloud-based VR platform for the crowdsourcing of embodied cognition with multiple human users and a virtual robot. The feasibility of the platform is evaluated from the latency of the interactive events and the motions of avatars, based on the number of users logging in to the VR scene and the distance between the local and server environments. We also discuss the feasibility of our system when dealing with large-scale datasets in embodied HRI experiments.

show abstract

Optimization of criterion for objective evaluation of HRI performance that approximates subjective evaluation: a case study in robot competition

Cited by 13 publications

References 24 publications

SIGVerse: A Cloud-Based VR Platform for Research on Multimodal Human-Robot Interaction

SIGVerse: A Cloud-Based VR Platform for Research on Multimodal Human-Robot Interaction

Learning Socially Appropriate Robo-waiter Behaviours through Real-time User Feedback

VR platform enabling crowdsourcing of embodied HRI experiments – case study of online robot competition

Contact Info

Product

Resources

About