Using multisensory cues for direction information in teleoperation: More is not always better

Benz, Tobias; Nitsch, Verena

doi:10.1109/icra.2017.7989773

Cited by 8 publications

(7 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Haptic guidance in the visual + haptic feedback condition was supposed to provide additional proprioceptive cues. However, our results suggest the visual + haptic feedback modality did not improve task performance, aligning with inferences made from prior studies that combining feedback modalities does not always lead to optimal performance (Benz and Nitsch, 2017).…”

Section: Discussionsupporting

confidence: 88%

“…One approach to providing feedback to surgeons is to employ multi-modal interfaces (Wickens et al, 2015) – using multiple sensory modalities to present information (Freeman et al, 2017) based on Wicken’s Multiple Resource Theory (MRT; Wickens, 2002). However, combining feedback modalities does not always lead to optimal performance (Benz and Nitsch, 2017); poorly presented or excessive feedback can be as bad as no feedback at all (Wickens et al, 2015). What is the optimal combination of feedback modalities (e.g., visual + haptic or visual + auditory or auditory + haptic or visual + auditory + haptic)?…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Neural Efficiency of Human–Robotic Feedback Modalities Under Stress Differs With Gender

Nuamah

Mantooth

Karthikeyan

et al. 2019

Front. Hum. Neurosci.

View full text Add to dashboard Cite

Sensory feedback, which can be presented in different modalities -single and combined, aids task performance in human-robotic interaction (HRI). However, combining feedback modalities does not always lead to optimal performance. Indeed, it is not known how feedback modalities affect operator performance under stress. Furthermore, there is limited information on how feedback affects neural processes differently for males and females and under stress. This is a critical gap in the literature, particularly in the domain of surgical robotics, where surgeons are under challenging socio-technical environments that burden them physiologically. In the present study, we posited operator performance as the summation of task performance and neurophysiological cost of maintaining that performance. In a within-subject design, we used functional near-infrared spectroscopy to assess cerebral activations of 12 participants who underwent a 3D manipulation task within a virtual environment with concurrent feedback (visual and visual + haptic) in the presence and absence of a cognitive stressor. Cognitive stress was induced with the serial-7 subtraction test. We found that while task performance was higher with visual than visual + haptic feedback, it degraded under stress. The two feedback modalities were found to be associated with varying neural activities and neural efficiencies, and these were stress-and genderdependent. Our findings engender further investigation into effectiveness of feedback modalities on males and females under stressful conditions in HRI.

show abstract

Section: Discussionsupporting

confidence: 88%

Section: Introductionmentioning

confidence: 99%

Neural Efficiency of Human–Robotic Feedback Modalities Under Stress Differs With Gender

Nuamah

Mantooth

Karthikeyan

et al. 2019

Front. Hum. Neurosci.

View full text Add to dashboard Cite

show abstract

“…Since the teleoperation system is built to serve the humans, it can explore the skill learning strategy from the view of human tutors to reduce the workload and improve the work efficiency. What is more, propelled by artificial intelligence technologies, such as deep learning [119, 120], reinforce learning [121], NN learning [122, 123], multimodal information perception [124, 125] etc., the level of autonomy of the teleoperation system can be greatly enhanced.…”

Section: Discussionmentioning

confidence: 99%

Combined perception, control, and learning for teleoperation: key technologies, applications, and challenges

Luo

Yang

2020

Cognitive Computation and Systems

View full text Add to dashboard Cite

Teleoperation provides a promising way for human-robot collaboration in the unknown or unstructured environments to perform a cooperative task. It enables humans to complete a task at a remote side and combines both the human's intelligence and the robots' capabilities in a collaborative task. Therefore, it is necessary to conduct cross researches in terms of robotics, artificial intelligence, sensors, and mechatronics. This study covers the state-of-the-arts research in terms of perception, control, and learning. In this review, key technologies about information perception mechanism, control algorithms, human-robot interface, and robot learning strategies for teleoperation are introduced. Then, a comprehensive survey is summarised in representative applications in teleoperation, and the existing challenges and potential directions of the development are discussed.

show abstract

“…However, increasing the number of modalities and com-bining them does not always result in increased situational awareness. The study by Benz and Nitsch [23] addressed the lack of applicable and accurate research for navigational tasks by designing 12 combinations of multimodal feedback for directional information, and further discussed the multimodal human-machine design implications for remote operated systems. The results of the study showed that combining modalities does not necessarily increase performance, but, on the contrary, may reduce the accuracy of the conveyed information.…”

Section: Multimodal Feedbackmentioning

confidence: 99%

“…Our research focuses on studying multimodal feedback in the context of delivering warnings to the remote operator. Whereas Benz and Nitsch [23] used simplified bird's-eye view and did not even attempt to simulate a realistic remote operation system in their experiment, we tested our hypothesis in a realistic simulator. We theorize that, in order to test out multimodal feedback effectively, the simulator should be sufficiently similar to a real-life scenarios, i.e., the feeling of presence should be high enough for the results to be applicable.…”

Section: Multimodal Feedbackmentioning

confidence: 99%

Multimodal Warnings in Remote Operation: The Case Study on Remote Driving

Kallioniemi

Burova

Mäkelä

et al. 2021

MTI

View full text Add to dashboard Cite

Developments in sensor technology, artificial intelligence, and network technologies like 5G has made remote operation a valuable method of controlling various types of machinery. The benefits of remote operations come with an opportunity to access hazardous environments. The major limitation of remote operation is the lack of proper sensory feedback from the machine, which in turn negatively affects situational awareness and, consequently, may risk remote operations. This article explores how to improve situational awareness via multimodal feedback (visual, auditory, and haptic) and studies how it can be utilized to communicate warnings to remote operators. To reach our goals, we conducted a controlled, within-subjects experiment in eight conditions with twenty-four participants on a simulated remote driving system. Additionally, we gathered further insights with a UX questionnaire and semi-structured interviews. Gathered data showed that the use of multimodal feedback positively affected situational awareness when driving remotely. Our findings indicate that the combination of added haptic and visual feedback was considered the best feedback combination to communicate the slipperiness of the road. We also found that the feeling of presence is an important aspect of remote driving tasks, and a requested one, especially by those with more experience in operating real heavy machinery.

show abstract

Using multisensory cues for direction information in teleoperation: More is not always better

Cited by 8 publications

References 40 publications

Neural Efficiency of Human–Robotic Feedback Modalities Under Stress Differs With Gender

Neural Efficiency of Human–Robotic Feedback Modalities Under Stress Differs With Gender

Combined perception, control, and learning for teleoperation: key technologies, applications, and challenges

Multimodal Warnings in Remote Operation: The Case Study on Remote Driving

Contact Info

Product

Resources

About