Augmented reality-aided tele-presence system for robot manipulation in industrial manufacturing

Peppoloni, Lorenzo; Brizzi, Filippo; Ruffaldi, Emanuele; Avizzano, Carlo Alberto

doi:10.1145/2821592.2821620

Cited by 18 publications

(12 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Manipulation of the observed space via AR was analyzed in robot teleoperation studies by Brizzi and Peppoloni et al in [69,70]. A headset-based AR environment was used to give a first-person perspective of the robot arm actuation, with AR cues added for robot arm's motion vectors.…”

Section: Improving Perception or Task Performance With Armentioning

confidence: 99%

“…the operated machinery is seen from outside, since first-person views i.e. views of the operated machinery from inside or ego-centric perspective have been covered in remote control teleoperation by [69,70]. The AR factor was varied by three Augmentation designs-a target-specific marker (A1), a 2D grid-map (A2) and a 3D scaling cube Fig.…”

Section: Experiments Designmentioning

confidence: 99%

“…In each test run, the participants were asked to navigate and accurately touch highlighted targets in the test area, using the remote control excavator's arm. This is a combination of the locomotion and pick-and-place tasks commonly used for depth estimation and positioning assessment [53,[69][70][71]76], adapted to the context of remotely controlling industrial machinery. The targets were highlighted in random order, and participants were scored on the number of targets reached.…”

Section: Experiments Designmentioning

confidence: 99%

See 2 more Smart Citations

Joint effects of depth-aiding augmentations and viewing positions on the quality of experience in augmented telepresence

et al. 2020

View full text Add to dashboard Cite

Virtual and augmented reality is increasingly prevalent in industrial applications, such as remote control of industrial machinery, due to recent advances in head-mounted display technologies and low-latency communications via 5G. However, the influence of augmentations and camera placement-based viewing positions on operator performance in telepresence systems remains unknown. In this paper, we investigate the joint effects of depth-aiding augmentations and viewing positions on the quality of experience for operators in augmented telepresence systems. A study was conducted with 27 non-expert participants using a real-time augmented telepresence system to perform a remote-controlled navigation and positioning task, with varied depth-aiding augmentations and viewing positions. The resulting quality of experience was analyzed via Likert opinion scales, task performance measurements, and simulator sickness evaluation. Results suggest that reducing the reliance on stereoscopic depth perception via camera placement has a significant benefit to operator performance and quality of experience. Conversely, the depth-aiding augmentations can partly mitigate the negative effects of inferior viewing positions. However the viewing-position based monoscopic and stereoscopic depth cues tend to dominate over cues based on augmentations. There is also a discrepancy between the participants' subjective opinions on augmentation helpfulness, and its observed effects on positioning task performance.

show abstract

Section: Improving Perception or Task Performance With Armentioning

confidence: 99%

Section: Experiments Designmentioning

confidence: 99%

Section: Experiments Designmentioning

confidence: 99%

See 1 more Smart Citation

Joint effects of depth-aiding augmentations and viewing positions on the quality of experience in augmented telepresence

et al. 2020

View full text Add to dashboard Cite

show abstract

“…As a new computer technology, it has been applied in various fields. In Reference [10], authors developed an augmented reality-aided tele-presence system for robot manipulation for the usage of industrial manufacturing, where a robot was successfully controlled remotely to pick a known object from a starting position and placing it in different target positions. In Reference [11], a telepresence system that allows the immsersive remote control of a working machine was developed, where an omnidirectional stereo vision head mounted on the machine to interact with the operator who wears a head-mounted display.…”

Section: Intruductionmentioning

confidence: 99%

An Augmented Reality Based Human-Robot Interaction Interface Using Kalman Filter Sensor Fusion

Fahmy

Sienz

2019

Sensors

View full text Add to dashboard Cite

In this paper, the application of Augmented Reality (AR) for the control and adjustment of robots has been developed, with the aim of making interaction and adjustment of robots easier and more accurate from a remote location. A LeapMotion sensor based controller has been investigated to track the movement of the operator hands. The data from the controller allows gestures and the position of the hand palm’s central point to be detected and tracked. A Kinect V2 camera is able to measure the corresponding motion velocities in x, y, z directions after our investigated post-processing algorithm is fulfilled. Unreal Engine 4 is used to create an AR environment for the user to monitor the control process immersively. Kalman filtering (KF) algorithm is employed to fuse the position signals from the LeapMotion sensor with the velocity signals from the Kinect camera sensor, respectively. The fused/optimal data are sent to teleoperate a Baxter robot in real-time by User Datagram Protocol (UDP). Several experiments have been conducted to test the validation of the proposed method.

show abstract

“…Within the field of robotics, natural 6-DOF end-effector control methods have been studied recently using motion tracking devices developed for augmented reality (AR) and virtual reality (VR). These approaches make use of hand tracking (Cancedda et al, 2017), wearable devices (Peppoloni et al, 2015), and VR controllers (Lipton et al, 2018; Rakita et al, 2017), showing feedback to the user in 3D user interfaces, AR, and VR interfaces (Cancedda et al, 2017; Lipton et al, 2018; Peppoloni et al, 2015). Motion capture techniques have been shown to be effective for pick-and-place and other manipulation actions due to the natural interaction of simply moving one’s arms, and the immersive qualities of AR and VR can aid in situational awareness.…”

Section: Related Workmentioning

confidence: 99%

Leveraging depth data in remote robot teleoperation interfaces for general object manipulation

Kent

Saldanha

Chernova

2019

The International Journal of Robotics Research

View full text Add to dashboard Cite

Robust remote teleoperation of high-degree-of-freedom manipulators is of critical importance across a wide range of robotics applications. Contemporary robot manipulation interfaces primarily utilize a free positioning pose specification approach to independently control each degree of freedom in free space. In this work, we present two novel interfaces, constrained positioning and point-and-click. Both novel approaches incorporate scene information from depth data into the grasp pose specification process, effectively reducing the number of 3D transformations the user must input. The novel interactions are designed for 2D image streams, rather than traditional 3D virtual scenes, further reducing mental transformations by eliminating the controllable camera viewpoint in favor of fixed physical camera viewpoints. We present interface implementations of our novel approaches, as well as free positioning, in both 2D and 3D visualization modes. In addition, we present results of a 90-participant user study evaluation comparing the effectiveness of each approach for a set of general object manipulation tasks, and the effects of implementing each approach in 2D image views versus 3D depth views. The results of our study show that point-and-click outperforms both free positioning and constrained positioning by significantly increasing the number of tasks completed and significantly reducing task failures and grasping errors, while significantly reducing the number of user interactions required to specify poses. In addition, we found that regardless of the interaction approach, the 2D visualization mode resulted in significantly better performance than the 3D visualization mode, with statistically significant reductions in task failures, grasping errors, task completion time, number of interactions, and user workload, all while reducing bandwidth requirements imposed by streaming depth data.

show abstract

Augmented reality-aided tele-presence system for robot manipulation in industrial manufacturing

Cited by 18 publications

References 13 publications

Joint effects of depth-aiding augmentations and viewing positions on the quality of experience in augmented telepresence

Joint effects of depth-aiding augmentations and viewing positions on the quality of experience in augmented telepresence

An Augmented Reality Based Human-Robot Interaction Interface Using Kalman Filter Sensor Fusion

Leveraging depth data in remote robot teleoperation interfaces for general object manipulation

Contact Info

Product

Resources

About