Understanding human motion and gestures for underwater human–robot collaboration

Abstract

In this paper, we present a number of robust methodologies for an underwater robot to visually detect, follow, and interact with a diver for collaborative task execution. We design and develop two autonomous diver-following algorithms, the first of which utilizes both spatial-and frequency-domain features pertaining to human swimming patterns in order to visually track a diver. The second algorithm uses a convolutional neural network-based model for robust tracking-by-detection. In addition, we propose a hand gesture-based human-robot communication framework that is syntactically simpler and computationally more efficient than the existing grammar-based frameworks. In the proposed interaction framework, deep visual detectors are used to provide accurate hand gesture recognition; subsequently, a finite-state machine performs robust and efficient gesture-to-instruction mapping. The distinguishing feature of this framework is that it can be easily adopted by divers for communicating with underwater robots without using artificial markers or requiring memorization of complex language rules. Furthermore, we validate the performance and effectiveness of the proposed methodologies through extensive field experiments in closed-and open-water environments. Finally, we perform a user interaction study to demonstrate the usability benefits of our proposed interaction framework compared to existing methods.

Fast Underwater Image Enhancement for Improved Visual Perception

“…Visually-guided AUVs (Autonomous Underwater Vehicles) and ROVs (Remotely Operated Vehicles) are widely used in important applications such as the monitoring of marine species migration and coral reefs [39], inspection of submarine cables and wreckage [5], underwater scene analysis, seabed mapping, human-robot collaboration [24], and more. One major operational challenge for these underwater robots is that despite using high-end cameras, visual sensing is often greatly affected by poor visibility, light refraction, absorption, and scattering [31,45,24]. These optical artifacts trigger non-linear distortions in the captured images, which severely affect the performance of visionbased tasks such as tracking, detection and classification, Input Generated (a) Perceptual enhancement of underwater images.…”

Person-following by autonomous robots: A categorical overview

“…Underwater missions are often conducted by a team of human divers and autonomous robots, who cooperatively perform a set of common tasks (Islam et al, 2018c; Sattar et al, 2008). The divers typically lead the tasks and interact with the robots, which follow the divers at certain stages of the mission (Islam et al, 2018a).…”

“…Deep learning-based object detection models have recently been investigated for underwater applications as well (Islam et al, 2018a; Shkurti et al, 2012). The state-of-the-art pre-trained models are typically trained (offline) on large underwater datasets and sometimes quantized or pruned in order to get faster inference by balancing robustness and efficiency (Islam et al, 2018a,c). As illustrated in Figure 8(f), once trained with sufficient data, these models are robust to noise and color distortions; additionally, a single model can be used to detect (and track) several objects at once.…”

“…Additionally, application-specific choices, such as whether to stay behind or side-by-side while following, the desired speed, and relevant anticipative behaviors (Granata and Bidaud, 2012; Mi et al, 2016) are essential considerations. Another important feature is to maintain the expected behavior during explicit interactions (Hu et al, 2014; Islam et al, 2018c), e.g., being stationary when the human is communicating, and exhibiting correct acknowledgement responses. These aspects of human–robot interaction are elaborately discussed in the following section.…”

“…Diver communicating instructions to autonomous underwater vehicle during a mission (Islam et al, 2018c). …”