Enhanced Teleoperation Using Autocomplete

Zein, Mohammad Kassem; Sidaoui, Abbas; Asmar, Daniel; Elhajj, Imad H.

doi:10.1109/icra40945.2020.9197140

Cited by 9 publications

(8 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A teleoperation system was developed in Zein et al (2020) that recognizes specific desired motions from the user joystick input and accordingly suggests to auto-complete the predicted motion through an augmented user interface. The proposed system was tested on Gazebo using a simulated Parrot Ar.…”

Section: Discussionmentioning

confidence: 99%

“…An “Autocomplete” framework was proposed in Zein et al (2020) that would support novice users in teleoperating complex systems such as drones. The system takes the human input from a Joystick, predicts what the actual desired teleoperation command is, and then shares it with the user through an augmented reality interface.…”

Section: Discussionmentioning

confidence: 99%

“…The potentials of integrating these two elements in robotics applications are manifold and provide a means of deciphering the traditional human-robot mismatch model. Specifically, in the context of human-robot collaboration, AI can be used to understand the real user intent filtered from the perceived tasks the robot traditionally performs as in the work of Zein et al (2020) . At the same time, AR can visualize information of the robot’s understanding of the user intent as in the work of Ghiringhelli et al (2014) , providing a closed feedback loop into the model mismatch paradigm.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

Bassyouni

Elhajj

2021

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

Recently, advancements in computational machinery have facilitated the integration of artificial intelligence (AI) to almost every field and industry. This fast-paced development in AI and sensing technologies have stirred an evolution in the realm of robotics. Concurrently, augmented reality (AR) applications are providing solutions to a myriad of robotics applications, such as demystifying robot motion intent and supporting intuitive control and feedback. In this paper, research papers combining the potentials of AI and AR in robotics over the last decade are presented and systematically reviewed. Four sources for data collection were utilized: Google Scholar, Scopus database, the International Conference on Robotics and Automation 2020 proceedings, and the references and citations of all identified papers. A total of 29 papers were analyzed from two perspectives: a theme-based perspective showcasing the relation between AR and AI, and an application-based analysis highlighting how the robotics application was affected. These two sections are further categorized based on the type of robotics platform and the type of robotics application, respectively. We analyze the work done and highlight some of the prevailing limitations hindering the field. Results also explain how AR and AI can be combined to solve the model-mismatch paradigm by creating a closed feedback loop between the user and the robot. This forms a solid base for increasing the efficiency of the robotic application and enhancing the user’s situational awareness, safety, and acceptance of AI robots. Our findings affirm the promising future for robust integration of AR and AI in numerous robotic applications.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

Bassyouni

Elhajj

2021

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition, the existing teleoperation systems can hardly meet the mental and physical demands as the operator needs to concentrate on every single detail in the control process instead of focusing on the task at hand [2]. To address these issues, autonomous teleoperation systems were developed in the existing literature [1]- [11]. The basic idea of autonomous teleoperation is to predict user intention and execute some parts of the task autonomously to decrease the demand on the operator and increase the task completion rate.…”

Section: Introductionmentioning

confidence: 99%

“…To better measure the performance, both objective metrics (e.g., task success rate and task completion time) and subjective metrics (e.g., operators' mental and physical demands) are considered. In some existing studies [1], [2], decision-making for mode-switching is assumed to be done by the operator. This approach brings an extra DoF to be controlled by the operator and generally introduces extra mental demand [4].…”

Section: Introductionmentioning

confidence: 99%

Wireless Control for Life‐Critical Actions

Kizilkaya¹,

Chang²,

Zhao³

et al. 2020

Engineering and Technology for Healthcare

View full text Add to dashboard Cite

Teleoperation can be very difficult due to limited perception, high communication latency, and limited degrees of freedom (DoFs) at the operator side. Autonomous teleoperation is proposed to overcome this difficulty by predicting user intentions and performing some parts of the task autonomously to decrease the demand on the operator and increase the task completion rate. However, decision-making for mode-switching is generally assumed to be done by the operator, which brings an extra DoF to be controlled by the operator and introduces extra mental demand. On the other hand, the communication perspective is not investigated in the current literature, although communication imperfections and resource limitations are the main bottlenecks for teleoperation. In this study, we propose an intelligent modeswitching framework by jointly considering mode-switching and communication systems. User intention recognition is done at the operator side. Based on user intention recognition, a deep reinforcement learning (DRL) agent is trained and deployed at the operator side to seamlessly switch between autonomous and teleoperation modes. A real-world data set is collected from our teleoperation testbed to train both user intention recognition and DRL algorithms. Our results show that the proposed framework can achieve up to 50% communication load reduction with improved task completion probability.

show abstract