The Helping Hand: An Assistive Manipulation Framework Using Augmented Reality and Tongue-Drive Interfaces

Recently, advancements in computational machinery have facilitated the integration of artificial intelligence (AI) to almost every field and industry. This fast-paced development in AI and sensing technologies have stirred an evolution in the realm of robotics. Concurrently, augmented reality (AR) applications are providing solutions to a myriad of robotics applications, such as demystifying robot motion intent and supporting intuitive control and feedback. In this paper, research papers combining the potentials of AI and AR in robotics over the last decade are presented and systematically reviewed. Four sources for data collection were utilized: Google Scholar, Scopus database, the International Conference on Robotics and Automation 2020 proceedings, and the references and citations of all identified papers. A total of 29 papers were analyzed from two perspectives: a theme-based perspective showcasing the relation between AR and AI, and an application-based analysis highlighting how the robotics application was affected. These two sections are further categorized based on the type of robotics platform and the type of robotics application, respectively. We analyze the work done and highlight some of the prevailing limitations hindering the field. Results also explain how AR and AI can be combined to solve the model-mismatch paradigm by creating a closed feedback loop between the user and the robot. This forms a solid base for increasing the efficiency of the robotic application and enhancing the user’s situational awareness, safety, and acceptance of AI robots. Our findings affirm the promising future for robust integration of AR and AI in numerous robotic applications.

show abstract

“…The authors in ( Chu et al, 2008 .) used a Tongue Drive System as input for an assistive grasping system facilitated through an AR interface.…”

Section: Discussionmentioning

confidence: 99%

“…The authors in (Chu et al, 2008.) used a Tongue Drive System as input for an assistive grasping system facilitated through an AR interface. The system implements the YOLO neural network [39] for object detection and a deep grasp algorithm for detecting the graspable locations for each object.…”

Section: Robotic Armsmentioning

confidence: 99%

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

Bassyouni

Elhajj

2021

Front. Robot. AI

View full text Add to dashboard Cite

show abstract

“…Especially when working with static robots, marker based approaches are widely used due to the simple setup and competitive accuracy. Several work including Aoki et al [14] or [15] et al relied on a marker based calibration between robot and AR device. However, Baratoff et al [16] stated, that the instrumentation of the real environment, by placing additional tools like markers, sensors or cameras is a main bottleneck for complex use cases especially in navigation that are working with dynamically changing environments.…”

Section: A Ar Calibration Methodsmentioning

confidence: 99%

A Markerless Deep Learning-based 6 Degrees of Freedom PoseEstimation for with Mobile Robots using RGB Data

Kästner¹,

Dimitrov²,

Lambrecht³

2020

Preprint

View full text Add to dashboard Cite

Augmented Reality has been subject to various integration efforts within industries due to its ability to enhance human machine interaction and understanding. Neural networks have achieved remarkable results in areas of computer vision, which bear great potential to assist and facilitate an enhanced Augmented Reality experience. However, most neural networks are computationally intensive and demand huge processing power thus, are not suitable for deployment on Augmented Reality devices. In this work we propose a method to deploy state of the art neural networks for real time 3D object localization on augmented reality devices. As a result, we provide a more automated method of calibrating the AR devices with mobile robotic systems. To accelerate the calibration process and enhance user experience, we focus on fast 2D detection approaches which are extracting the 3D pose of the object fast and accurately by using only 2D input. The results are implemented into an Augmented Reality application for intuitive robot control and sensor data visualization. For the 6D annotation of 2D images, we developed an annotation tool, which is, to our knowledge, the first open source tool to be available. We achieve feasible results which are generally applicable to any AR device thus making this work promising for further research in combining high demanding neural networks with Internet of Things devices.

show abstract

“…Furthermore, a combination of sensing modalities, learning methods, and control schemes are used by many researchers to create more efficient HRI systems. Chu et al created a framework that used a tongue-driven interface to obtain the user's intent and verify it visually through an augmented reality headset [46]. Sasaki et al developed a novel tongue interface based on the classification of tongue motions from the surface EMG signals of the suprahyoid muscles detected at the underside of the jaw [47].…”

Section: Assistive Robotic Manipulatorsmentioning

confidence: 99%

Human-Robot Interaction in Rehabilitation and Assistance: a Review

Mohebbi

2020

Curr Robot Rep

View full text Add to dashboard Cite

Purpose of Review Research in assistive and rehabilitation robotics is a growing, promising, and challenging field emerged due to various social and medical needs such as aging populations, neuromuscular, and musculoskeletal disorders. Such robots can be used in various day-today scenarios or to support motor functionality, training, and rehabilitation. This paper reflects on the human-robot interaction perspective in rehabilitation and assistive robotics and reports on current issues and developments in the field. Recent Findings The survey on the literature reveals that new efforts are put on utilizing machine learning approaches alongside novel developments in sensing technology to adapt the systems with user routines in terms of activities for assistive systems and exercises for rehabilitation devices to fit each user's need and maximize their effectiveness. Summary A review of recent research and development efforts on human-robot interaction in assistive and rehabilitation robotics is presented in this paper. First, different subdomains in assistive and rehabilitation robotic research are identified, and accordingly, a survey on the background and trends of such developments is provided.

show abstract

The Helping Hand: An Assistive Manipulation Framework Using Augmented Reality and Tongue-Drive Interfaces

Cited by 23 publications

References 24 publications

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

Augmented Reality Meets Artificial Intelligence in Robotics: A Systematic Review

A Markerless Deep Learning-based 6 Degrees of Freedom PoseEstimation for with Mobile Robots using RGB Data

Human-Robot Interaction in Rehabilitation and Assistance: a Review

Contact Info

Product

Resources

About