Surface-Type Classification Using RGB-D

Abstract

This paper proposes an approach to improve surface-type classification of images containing inconsistently illuminated surfaces. When a mobile inspection robot is visually inspecting surface-types in a dark environment and a directional light source is used to illuminate the surfaces, the images captured may exhibit illumination variance that can be caused by the orientation and distance of the light source relative to the surfaces. In order to accurately classify the surface-types in these images, either the training image dataset needs to completely incorporate the illumination variance or a way to extract colour features that can provide high classification accuracy needs to be identified. In this paper diffused reflectance values are extracted as new colour features to classifying surface-types. In this approach, RGB-D data is collected from the environment, and a reflectance model is used to calculate a diffused reflectance value for a pixel in each Red, Green, Blue (RGB) colour channel. The diffused reflectance values can be used to train a multi-class support vector machine classifier to classify surface-types. Experiments are conducted in a mock bridge maintenance environment using a portable RGB-Depth (RGB-D) sensor package with an attached light source to collect surface-type data. The performance of a classifier trained with diffused reflectance values is compared against classifiers trained with other colour features including RGB and L*a*b* colour spaces. Results show that the classifier trained with the diffused reflectance values can achieve consistently higher classification accuracy than the classifiers trained with RGB and L*a*b* features. For test images containing a single surface plane, diffused reflectance values consistently provide greater than 90% classification accuracy; and for test images containing a complex scene with multiple surface-types and surface planes, diffused reflectance values are shown to provide an increase in overall accuracy over RGB and L*a*b* by 49.24% and 13.66%, respectively. Note to Practitioners: This paper was motivated by the problem of inspecting inconsistently illuminated steel surfaces on a bridge structure using a robot manipulator. Existing approaches for colour-based surface classification are susceptible to illumination variance. This paper proposes the use of diffused reflectance values, which combines the use of colour and depth data to improve accuracy. In this approach, the diffused reflectance values of each image pixel are calculated by using the distance and angle between the surface represented by a pixel and the light source. The diffused reflectance values are calculated in each colour channel (Red, Green, Blue) to provide three features to classify different surface-types. This proposed approach can be applied to surface classification tasks where the light source does not uniformly illuminate the scene in the image.

Model-based reinforcement learning approach for deformable linear object manipulation

“…The details are discussed in Section IV-C In our experiment, a Baxter, dual-arm industry robot, acted as the Rope Manipulator. While a Kinect2 sensor was used as the Control Point Detector based upon a method from [16]. A workstation took the role of the Central PC that was responsible for state generation, policy searches and robot control.…”

A Tradeoff Analysis of a Cloud-Based Robot Navigation Assistant Using Stereo Image Processing

“…Nevertheless, there is always the possibility of combining the visual processed information with that of more basic and reactive sensors. Besides, RGB-D cameras are being developed quickly and at relatively low prices [31]. However, low priced RGB-D cameras are currently aimed for the game market, and, hence, they present several disadvantages.…”

Experimental Evaluation of Nearest Neighbor Exploration Approach in Field Environments

“…The results of such inspections are subjective, as different human investigators will conceivably give different ratings to the same surface [7]. To eliminate the risk to people and to have objective ratings of surface condition, it is useful to be able to deploy robots into an environment and have them collect the required information autonomously [8]. The advantage of autonomous robots could be seen during the Fukushima nuclear plant crisis in 2011 when a tethered robot's communications cable became caught and was pulled out of its socket.…”