A Deep Learning-Based Dirt Detection Computer Vision System for Floor-Cleaning Robots with Improved Data Collection

Abstract

Floor-cleaning robots are becoming increasingly more sophisticated over time and with the addition of digital cameras supported by a robust vision system they become more autonomous, both in terms of their navigation skills but also in their capabilities of analyzing the surrounding environment. This document proposes a vision system based on the YOLOv5 framework for detecting dirty spots on the floor. The purpose of such a vision system is to save energy and resources, since the cleaning system of the robot will be activated only when a dirty spot is detected and the quantity of resources will vary according to the dirty area. In this context, false positives are highly undesirable. On the other hand, false negatives will lead to a poor cleaning performance of the robot. For this reason, a synthetic data generator found in the literature was improved and adapted for this work to tackle the lack of real data in this area. This synthetic data generator allows for large datasets with numerous samples of floors and dirty spots. A novel approach in selecting floor images for the training dataset is proposed. In this approach, the floor is segmented from other objects in the image such that dirty spots are only generated on the floor and do not overlap those objects. This helps the models to distinguish between dirty spots and objects in the image, which reduces the number of false positives. Furthermore, a relevant dataset of the Automation and Control Institute (ACIN) was found to be partially labelled. Consequently, this dataset was annotated from scratch, tripling the number of labelled images and correcting some poor annotations from the original labels. Finally, this document shows the process of generating synthetic data which is used for training YOLOv5 models. These models were tested on a real dataset (ACIN) and the best model attained a mean average precision (mAP) of 0.874 for detecting solid dirt. These results further prove that our proposal is able to use synthetic data for the training step and effectively detect dirt on real data. According to our knowledge, there are no previous works reporting the use of YOLOv5 models in this application.

Vision-based dirt distribution mapping using deep learning

“…For training the proposed model, we combined publicly available dirt datasets with our own new data to produce a dataset to be used. The work 33 proved the effectiveness of using synthetic data to train a dirt detection model. Thus, we used the synthetically generated dataset provided in 33 as a part of our dataset.…”

“…The work 33 proved the effectiveness of using synthetic data to train a dirt detection model. Thus, we used the synthetically generated dataset provided in 33 as a part of our dataset. We also combined the ACIN dataset 34 as it consists of real images with severe lighting conditions, complex floor patterns and blurred images to provide a more robust dataset.…”

“…Furthermore, as the ACIN dataset only consisted of a few hundred images, we also collected our own data with an RGB camera across various floor patterns, angles and lighting conditions to create a well-balanced dataset. After combining our dataset with the publicly available datasets (i.e., synthetic dataset in 33 and ACIN dataset), we applied data augmentations techniques such as flipping and rotations to diversify our dataset further. This augmentation can reduce the CNN learning rate as well as reduces over fitting.…”

“…A multitude of research has been performed to classify the dirt types available on a floor to improve the performance of cleaning robots. Among these, developments of systems for mud and dirt separation 18 , liquid and solid dirt separation 19 , stain detection 20 , and dirt type classification of office environments 21 could be found. However, the scope of much of the work is limited to the classification of dirt, and the creation of dirt distribution maps to be used by a set of cleaning robots with heterogeneous capabilities has not been discussed.…”