Multimodal sensor-based whole-body control for human–robot collaboration in industrial settings

Symbiotic human-robot collaborative assembly

“…The vision/force integration is also explored in the context of collaborative screw fastening [40], where the data from Kinect, black/white camera and force sensor, deployed to track human hand, screw and contact force, respectively, are used alternately for robot control. De Gea Fernández et al [62] extended sensor data integration from IMU, RGB-D (red, green, blue, depth) camera and laser scanner to robot whole-body control. The RGB-D and laser scanner are responsible for human tracking while the IMU, integrated into the operator's clothes, recognises the human intention through gestures.…”

Embedded multimodal interfaces in robotics: applications, future trends, and societal implications

“…In traditional robot applications a violation of these borders just results in an alarm and a shutdown of the production line. In more advanced applications this information is used to adapt the behavior of the robot in case the boundaries mentioned above are violated by an object or a human [de Gea Fernández et al 2017].…”

“…Safety boundaries are not static in modern robotics, but can be adaptive and vary with the context of the robot's task and application [Vogel et al 2013], e.g. the robot would not go to a full stop but rather slow to a predefined speed in a human-robot cooperation scenario [de Gea Fernández et al 2017]. Because the kind of adaptation of the robot is dependent on the context of the task, applying this adaptive type of safety level can also be seen as a context or application-specific safety level (see also Haddadin [2015]).…”

“…Hence, advanced interfaces that make use of multimodal data to enable explicit and implicit interaction with humans (see Section 14.3) must not only focus on establishing a representation of the user state but must also encompass a description of the status of the robot. There are simple mechanisms to present the state of the robot to a human operator, e.g., written or colored light feedback [de Gea Fernández et al 2017], video data feedback from the point of view of the robot (i.e., its internal cameras), 3D reconstruction of the robot in its environment, or force feedback which conveys an impression of the robot's tactile perception during its interaction with the environment. These modes of feedback are able to communicate information about the internal state of the robot.…”

“…For example, a human must be detected when he or she has penetrated the work space of an industrial robot. Different sensors on the robot or in the environment, such as lasers, 3D depth-image camera, or motion sensors (that are attached to the human or human's clothes) can be combined to detect the location of the person and predict their path [de Gea Fernández et al 2017]. Furthermore, the weaknesses of one sensor, i.e., a restricted range, can be compensated by another to improve the recognition of a human entering the work space.…”