Intuitive Bare-Hand Teleoperation of a Robotic Manipulator Using Virtual Reality and Leap Motion

Jang, Inmo; Carrasco, Joaquín; Weightman, Andrew; Lennox, Barry

doi:10.1007/978-3-030-25332-5_25

Cited by 27 publications

(12 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, the data collection for building LaSeSOM is described first, and afterwards the framework is used to present different representation results in a robotic teleoperated soft object manipulation task via Leap Motion [33] demonstration.…”

Section: Resultsmentioning

confidence: 99%

LaSeSOM: A Latent and Semantic Representation Framework for Soft Object Manipulation

Zhou

Zhu

Huo

et al. 2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Soft object manipulation has recently gained popularity within the robotics community due to its potential applications in many economically important areas. Although great progress has been recently achieved in these types of tasks, most state-of-the-art methods are case-specific; They can only be used to perform a single deformation task (e.g. bending), as their shape representation algorithms typically rely on "hard-coded" features. In this paper, we present LaSeSOM, a new feedback latent representation framework for semantic soft object manipulation. Our new method introduces internal latent representation layers between low-level geometric feature extraction and high-level semantic shape analysis; This allows the identification of each compressed semantic function and the formation of a valid shape classifier from different feature extraction levels. The proposed latent framework makes soft object representation more generic (independent from the object's geometry and its mechanical properties) and scalable (it can work with 1D/2D/3D tasks). Its high-level semantic layer enables to perform (quasi) shape planning tasks with soft objects, a valuable and underexplored capability in many soft manipulation tasks. To validate this new methodology, we report a detailed experimental study with robotic manipulators.

show abstract

Section: Resultsmentioning

confidence: 99%

LaSeSOM: A Latent and Semantic Representation Framework for Soft Object Manipulation

Zhou

Zhu

Huo

et al. 2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

show abstract

“…Leap Motion Interaction Engine) that facilitate to interact with virtual objects using bare hands without any hand-held equipment. In our previous work [19], hands sensed by LM are reasonably accurate and much more natural to use compared with the use of hand-held devices.…”

Section: Methodology: Omnipotent Virtual Giantmentioning

confidence: 95%

Omnipotent Virtual Giant for Remote Human-Swarm Interaction

Jang

Arvin

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

This paper proposes an intuitive human-swarm interaction framework inspired by our childhood memory in which we interacted with living ants by changing their positions and environments as if we were omnipotent relative to the ants. In virtual reality, analogously, we can be a super-powered virtual giant who can supervise a swarm of mobile robots in a vast and remote environment by flying over or resizing the world, and coordinate them by picking and placing a robot or creating virtual walls. This work implements this idea by using Virtual Reality along with Leap Motion, which is then validated by proof-of-concept experiments using real and virtual mobile robots in mixed reality. We conduct a usability analysis to quantify the effectiveness of the overall system as well as the individual interfaces proposed in this work. The results revealed that the proposed method is intuitive and feasible for interaction with swarm robots, but may require appropriate training for the new end-user interface device.

show abstract

“…Since VR technology has increased in fidelity and reduced in cost over recent years, it has been investigated as a medium for interfaces for robot supervision and control. Jang et al use VR for virtual kinesthetic teaching of a robot arm ( Jang et al, 2019 ), ( Jang et al, 2021 ) and discuss the advantages of a system that integrates ROS, Unity and Virtual Reality. Whitney et al describe an interface for connecting two ROS controlled robot arms in VR and remotely operating them with sensor data ( Whitney et al, 2018 ).…”

Section: Related Workmentioning

confidence: 99%

Exploring Effects of Information Filtering With a VR Interface for Multi-Robot Supervision

2021

View full text Add to dashboard Cite

Supervising and controlling remote robot systems currently requires many specialised operators to have knowledge of the internal state of the system in addition to the environment. For applications such as remote maintenance of future nuclear fusion reactors, the number of robots (and hence supervisors) required to maintain or decommission a facility is too large to be financially feasible. To address this issue, this work explores the idea of intelligently filtering information so that a single user can supervise multiple robots safely. We gathered feedback from participants using five methods for teleoperating a semi-autonomous multi-robot system via Virtual Reality (VR). We present a novel 3D interaction method to filter the displayed information to allow the user to read information from the environment without being overwhelmed. The novelty of the interface design is the link between Semantic and Spatial filtering and the hierarchical information contained within the multi robot system. We conducted a user study including a cohort of expert robot teleoperators comparing these methods; highlighting the significant effects of 3D interface design on the performance and perceived workload of a user teleoperating many robot agents in complex environments. The results from this experiment and subjective user feedback will inform future investigations that build upon this initial work.

show abstract

Intuitive Bare-Hand Teleoperation of a Robotic Manipulator Using Virtual Reality and Leap Motion

Cited by 27 publications

References 15 publications

LaSeSOM: A Latent and Semantic Representation Framework for Soft Object Manipulation

LaSeSOM: A Latent and Semantic Representation Framework for Soft Object Manipulation

Omnipotent Virtual Giant for Remote Human-Swarm Interaction

Exploring Effects of Information Filtering With a VR Interface for Multi-Robot Supervision

Contact Info

Product

Resources

About