Natural teaching for humanoid robot via human-in-the-loop scene-motion cross-modal perception

Abstract

Purpose This paper aims to present a human-in-the-loop natural teaching paradigm based on scene-motion cross-modal perception, which facilitates the manipulation intelligence and robot teleoperation. Design/methodology/approach The proposed natural teaching paradigm is used to telemanipulate a life-size humanoid robot in response to a complicated working scenario. First, a vision sensor is used to project mission scenes onto virtual reality glasses for human-in-the-loop reactions. Second, motion capture system is established to retarget eye-body synergic movements to a skeletal model. Third, real-time data transfer is realized through publish-subscribe messaging mechanism in robot operating system. Next, joint angles are computed through a fast mapping algorithm and sent to a slave controller through a serial port. Finally, visualization terminals render it convenient to make comparisons between two motion systems. Findings Experimentation in various industrial mission scenes, such as approaching flanges, shows the numerous advantages brought by natural teaching, including being real-time, high accuracy, repeatability and dexterity. Originality/value The proposed paradigm realizes the natural cross-modal combination of perception information and enhances the working capacity and flexibility of industrial robots, paving a new way for effective robot teaching and autonomous learning.

Multimode fusion perception for transparent glass recognition

“…The combination of different attributes constitutes the diversity and particularity of objects. For robots, the general recognition of object attributes only realizes robotic basic perception (Xu et al , 2019; Shaik and Rufus, 2021). With the increase in environment diversification and application tasks, this low-level perception makes it difficult to meet higher demands, such as clothes category (Fang et al , 2018) just judging functions but cloth attributes (fabric, hand feeling, weight) (Yuan et al , 2018) providing the selection of comfort.…”

Human–robot shared control for humanoid manipulator trajectory planning

“…Recently, the research and application of human–robot intelligent collaboration in various fields are greatly promoted (Miller and Ju, 2015), such as robots for rehabilitation (Mao and Agrawal, 2012; Pehlivan et al , 2015; Powell and O’Malley, 2012), rescue (Passenberg et al , 2010) and teleoperation (Birk et al , 2018; Bouteraa and Ben Abdallah, 2017; Li and Ge, 2014; Xu et al , 2019). Humans and robots have complementary capabilities, and in many cases, their collaboration is essential.…”

An integrated vision-based system for efficient robot arm teleoperation

“…Although Denavit-Hartenberg modeling (D-H modeling) is already a mature method in describing forward robot kinematics (Zhu et al, 2020), it would face extra problems when solving the inverse kinematics, such as the existence of solutions and multiply solutions. Based on this fact, commonly used approaches focus more on using certain limb's geometric relationships to obtain the joint angles (Ju et al, 2014;Reddivari et al, 2014;Xu et al, 2019). However, the main drawbacks of these geometric metrics are that they require extra patience and a good sense of space to prevent possible mismatches caused by human error.…”