Human-Aware Collaborative Robots in the Wild: Coping with Uncertainty in Activity Recognition

Abstract

This study presents a novel approach to cope with the human behaviour uncertainty during Human-Robot Collaboration (HRC) in dynamic and unstructured environments, such as agriculture, forestry, and construction. These challenging tasks, which often require excessive time, labour and are hazardous for humans, provide ample room for improvement through collaboration with robots. However, the integration of humans in-the-loop raises open challenges due to the uncertainty that comes with the ambiguous nature of human behaviour. Such uncertainty makes it difficult to represent high-level human behaviour based on low-level sensory input data. The proposed Fuzzy State-Long Short-Term Memory (FS-LSTM) approach addresses this challenge by fuzzifying ambiguous sensory data and developing a combined activity recognition and sequence modelling system using state machines and the LSTM deep learning method. The evaluation process compares the traditional LSTM approach with raw sensory data inputs, a Fuzzy-LSTM approach with fuzzified inputs, and the proposed FS-LSTM approach. The results show that the use of fuzzified inputs significantly improves accuracy compared to traditional LSTM, and, while the fuzzy state machine approach provides similar results than the fuzzy one, it offers the added benefits of ensuring feasible transitions between activities with improved computational efficiency.

Table-Balancing Cooperative Robot Based on Deep Reinforcement Learning

“…Additionally, researchers have proposed various HRI techniques [18][19][20][21][22][23][24][25][26][27]. These preexisting HRI techniques span from empirical social rules [18][19][20] to an end-to-end learning framework [21][22][23]26,27].…”

“…However, the process of creating datasets for learning includes costly expert timing for when to talk. For human activity recognition (HAR), several methods are proposed [24,25]. Beril Yalçinkaya et al [24] enhanced the predictability of human activity sequences by combining long short-term memory (LSTM) and fuzzy logic.…”

“…For human activity recognition (HAR), several methods are proposed [24,25]. Beril Yalçinkaya et al [24] enhanced the predictability of human activity sequences by combining long short-term memory (LSTM) and fuzzy logic. Imran Ullah Khan et al [25] developed a hybrid model by incorporating a convolutional neural network (CNN) and LSTM for activity recognition, where the CNN is used for spatial feature extraction and the LSTM network is utilized for learning temporal information.…”

“…In most previous studies using RL, robots perform tasks on their own on behalf of humans [8][9][10][11][12][13][14][15][16][17]. Various human-robot interaction (HRI) technologies [18][19][20][21][22][23][24][25][26][27] are being studied, but most of them focus on research for robots to engage in conversation with humans [18][19][20][21][22][23] or utilize human feedback for robot learning [26,27] rather than collaborating with humans to perform tasks. In the industry, cobots (collaborative robots) are robots that cooperate with humans without separating barriers, but most of them are fixed in specific places.…”