Gesture- and vision-based automatic grasping and flexible placement in teleoperation

Abstract

Teleoperation system has attracted a lot of attention because of its advantages in dangerous or unknown environment.It is very difficult to develop an operating system that can complete complex tasks in an completely autonomous. This paper proposes a robot arm control strategy based on gesture and visual perception. The strategy combines the advantages of humans and robots to obtain a convenient and flexible interaction model. The hand data were obtained by Leap-Motion. Then a neural network algorithm was used to classify the nine gestures used for robot control by a finite state machine. The control mode switched between indicative control and mapping control. The robot acquired a autonomous grasp ability by incorporating YOLO 6D, depth data, and a probabilistic roadmap planner algorithm.The robot completed most of the trajectory independently, and a few flexible trajectories required a user to make mapping actions. This interactive mode reduces the burden of the user to a certain extent, that makes up for the shortcomings of traditional teleoperation.

Latency mitigation using applied HMMs for mixed reality-enhanced intuitive teleoperation in intelligent robotic welding

“…Teleoperation systems for remote welding have the potential to save humans from operating in risky, high-hazard situations and environments, such as remote sites like deserts, combat situations, areas of natural disaster, underserved sites [1]- [3], or hyper/hypobaric conditions, such as high-altitude and underwater environments [4]- [6]. Equally, teleoperation enables the most productive utilization of scarce expertise.…”