Simultaneous exploration and segmentation for search and rescue

Abstract

We consider the problem of active victim segmentation during a search‐and‐rescue (SAR) exploration mission. The robot is equipped with a multimodal sensor suite consisting of a camera, lidar, and pan‐tilt thermal sensor. The robot enters an unknown scene, builds a 3D model incrementally, and the proposed method simultaneously (a) segments the victims from incomplete multimodal measurements and (b) controls the motion of the thermal camera. Both of these tasks are difficult due to the lack of natural training data and the limited number of real‐world trials. In particular, we overcome the absence of training data for the segmentation task by employing a manually designed generative model, which provides a semisynthetic training data set. The limited number of real‐world trials is tackled by self‐supervised initialization and optimization‐based guiding of the motion control learning. In addition to that, we provide a quantitative evaluation of the proposed method on several real testing scenarios using the real SAR robot. Finally, we also provide a data set which will allow for further development of algorithms on the real data.

Thermal, Multispectral, and RGB Vision Systems Analysis for Victim Detection in SAR Robotics

“…Likewise, almost all works focus on integrating sensors in UAVs from the top view plane, working on datasets. Along these lines, references [22,38] propose a different approach based on detecting and localizing victims online from UGVs in navigation areas with low-light conditions. Table 1 provides a detailed overview of the most relevant works and presents a comparison concerning the methods and techniques employed in their development.…”

The UMA-SAR Dataset: Multimodal data collection from a ground vehicle during outdoor disaster response training exercises

“…Among these representative categories, the localization of victims in a disaster site in life-critical SAR activities is highly important (Delmerico et al, 2019). This search of potential victims can be performed using multi-spectral images (Petříček et al, 2019) and including human body parts discovery (Oliveira et al, 2018). The aforementioned research topics are aimed at the disaster site recognition and mapping, where the location of first-response staff (and more information such as emergency vehicles, civilians, rubble mounds, tents, and victims) is relevant for situational awareness, not only useful for robotic systems, but also for coordination personnel in charge of SAR missions.…”

“…In some cases, the lack of suitable datasets can be partially mitigated by computing synthetic images, as in the DISC dataset (Jeon et al, 2019), which consists of sufficiently realistic stereo renderings of fire and collapsing structures that allow comparing before and after disaster conditions. In this sense, Petříček et al (2019) created a semisynthetic dataset consisting of chroma key images of human victim stand-ins that can be overlaid onto background images of disaster environments. Precisely, recent datasets have incorporated images of human figures for intelligent victim detection.…”

“…In addition to visible light images, thermal images can be useful for scene understanding in texture-less, dark, and smoke- or dust-filled SAR settings. Petříček et al (2019) addressed the combination of thermal and visual images for multimodal victim detection from a robotic vehicle in mock-up disaster sites. Their dataset documents the research goal of controlling the pan-and-tilt of a thermal camera for human body segmentation of a 3D occupancy grid built from motion estimations, panoramic RGB images and depth information.…”

“…In addition, perception in disaster situations poses distinctive challenges owing to limited image quality, a diversity of target objects, unstructured search environments, and integration of visual information with other data modalities (Arnold et al, 2019). Thus, the availability of representative datasets becomes essential for developing, training, testing, and benchmarking new artificial intelligence solutions for perception, classification, localization, and mapping in the challenging disaster response application domain (Bañuls et al, 2020; Dubé et al, 2020; Petříček et al, 2019).…”