Human–Robot Collaboration in 3D via Force Myography Based Interactive Force Estimations Using Cross-Domain Generalization

Abstract

In this study, human robot collaboration (HRC) via force myography (FMG) bio-signal was investigated. Interactive hand force was estimated during moving a wooden rod in 3D with a Kuka robot. A baseline FMG-based deep convolutional neural network (FMG-DCNN) model could moderately estimate applied forces during the HRC task. Model performance can be improved with additional training data; however, collection of it was impractical and time-consuming. Available long-term multiple source data (32 feature spaces) during human robot interaction (HRI) with a linear robot collected over a long time period might be useful. Therefore, we explored a cross-domain generalization (CDG) technique that allowed pretraining a model to transfer knowledge between two unrelated source (2D-HRI) and target data (3D-HRC) for the first time. An FMG-based transfer learning with CDG (TL-CDG) model trained with these multiple source domains was examined in estimating applied forces from 16-channel FMG data during interactions with the Kuka robot. Two target scenarios were evaluated: case i) collaborative task of moving the wooden rod in 3D, and case ii) grasping interactions in 1D. In both cases, few calibration data finetuned the TL-CDG model and improved recognizing out-of-domain target data (case i: R 2 ≈60-63%, and case ii: R 2 ≈79-87%) compared to the baseline FMG-DCNN model. Hence, cross-domain generalization could be useful in platform-independent FMG-based HRI applications.

Detecting Safety Anomalies in pHRI Activities via Force Myography

“…For compliant collaboration during training and testing, motion trajectories were not fixed between a start and end point; rather, it was bounded by 6-axis rectangular areas in the 1D, 2D, and 3D workspaces [ 51 ]. This allowed flexible dynamic motions within certain ranges, instead of restricted movements.…”

“…This control mechanism could be implemented to control and monitor large industrial assembly line robots that do not have any joint torque sensors to recognize the external environment. Further description of this pHRI platform can be found in [ 51 ].…”

“…Common industrial tasks such as object transportation or handover may require the worker to apply hand forces during interactions with the robot. Estimating the applied hand force to interact with a robot in dynamic motions using FMG signals was found favorable in physical interactions [ 48 , 49 , 50 , 51 , 52 , 53 ]. These studies implemented compliant collaboration, where a robot’s trajectory was dictated by the applied hand force of a human worker.…”

Unsupervised, Semi-Supervised Interactive Force Estimations During pHRI via Generated Synthetic Force Myography Signals

“…In the literature, CNN based GAN models are more dominant and have shown good performances in image classifications and pattern recognition. In our previous studies, we found that the model could effectively learn discriminative features of time-series interactive data [16], [17]. In this study, window size of the FMG-DCGAN model was set to 10ms evaluate the smallest time-series input (every single row) in the system.…”

“…The study showed that intrasession models could estimate applied forces well (accuracies of 94%>R 2 >82%) where training and evaluation were carried out in the same session, had similar feature distributions, and were labeled properly. Recently, adapting domain adaptation, domain generalization and cross-domain generalization techniques enabled an SVR, and a convolutional neural network (CNN) model trained with labeled FMG data from multiple pHRI source domains to recognize applied hand forces in unknown pHRI target scenarios that the model had never seen before [15]- [17]. Finetuning allowed required periodic calibrations to recognize any unseen, instantaneous signals.…”