An optical flow-based method for condition-based maintenance and operational safety in autonomous cleaning robots

Self Cite

An automated Condition Monitoring (CM) and real-time controlling framework is essential for outdoor mobile robots to ensure the robot’s health and operational safety. This work presents a novel Artificial Intelligence (AI)-enabled CM and vibrotactile haptic-feedback-based real-time control framework suitable for deploying mobile robots in dynamic outdoor environments. It encompasses two sections: developing a 1D Convolutional Neural Network (1D CNN) model for predicting system degradation and terrain flaws threshold classes and a vibrotactile haptic feedback system design enabling a remote operator to control the robot as per predicted class feedback in real-time. As vibration is an indicator of failure, we identified and separated system- and terrain-induced vibration threshold levels suitable for CM of outdoor robots into nine classes, namely Safe, moderately safe system-generated, and moderately safe terrain-induced affected by left, right, and both wheels, as well as severe classes such as unsafe system-generated and unsafe terrain-induced affected by left, right, and both wheels. The vibration-indicated data for each class are modelled based on two sensor data: an Inertial Measurement Unit (IMU) sensor for the change in linear and angular motion and a current sensor for the change in current consumption at each wheel motor. A wearable novel vibrotactile haptic feedback device architecture is presented with left and right vibration modules configured with unique haptic feedback patterns corresponding to each abnormal vibration threshold class. The proposed haptic-feedback-based CM framework and real-time remote controlling are validated with three field case studies using an in-house-developed outdoor robot, resulting in a threshold class prediction accuracy of 91.1% and an effectiveness that, by minimising the traversal through undesired terrain features, is four times better than the usual practice.

Section: Related Workmentioning

confidence: 99%

AI-Enabled Vibrotactile Feedback-Based Condition Monitoring Framework for Outdoor Mobile Robots

Pookkuttath,

Abdulkader,

Elara

et al. 2023

Self Cite

“…Firstly, there was an IMU sensor-based study [18] where the vibration-affected change in linear acceleration, angular velocity, and angular acceleration of the robot was modeled as vibration data. In the second work, a monocular camera sensor [19] was used, and the vibration was modeled as sparse optical flow. Here, the change in optical flow vector displacement over consecutive frames due to the robot's vibrations are used as vibration indication data for training different abnormal vibration source classes.…”

Section: Related Workmentioning

confidence: 99%

“…Convolutional Neural Networks can be used based on process data such as 1D signal, 2D image, and 3D video [35][36][37][38][39]. In the proposed study, a 1D Convolutional Neural Network (1D CNN) model is adopted to classify the vibration threshold levels and for real-time prediction due to its simple structure and low computation cost, as well as from considering previous CM work [18,19] results compared with other AI models. A four-layer 1D CNN model is structured, and it works based on the convolution operations on data vectors, as mentioned in Equations ( 2) and (3) [40].…”

Section: One-dimensional Cnn Modeling For Threshold Level Classificationmentioning

confidence: 99%

“…Our research focuses on utilizing the typical onboard sensors for CM applications, other than their usual uses for perception, navigation, obstacle avoidance, path planning, etc. Towards this research objective, we have been developing suitable CM frameworks for indoor mobile robots and presented the works based on an IMU sensor [18] and camera [19]. Developing an appropriate CM framework for an outdoor mobile robot is more challenging considering the hostile, dynamic, unpredictable, and highly uneven terrain and climate changes in the outdoor workspace environment, especially for fumigation or fogging, lawn mowing, and pavement sweeping applications.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

AI-Enabled Condition Monitoring Framework for Outdoor Mobile Robots Using 3D LiDAR Sensor

Pookkuttath,

Palanisamy,

Elara

2023

Self Cite

An automated condition monitoring (CM) framework is essential for outdoor mobile robots to trigger prompt maintenance and corrective actions based on the level of system deterioration and outdoor uneven terrain feature states. Vibration indicates system failures and terrain abnormalities in mobile robots; hence, five vibration threshold classes for CM in outdoor mobile robots were identified, considering both vibration source system deterioration and uneven terrain. This study proposes a novel CM approach for outdoor mobile robots using a 3D LiDAR, employed here instead of its usual use as a navigation sensor, by developing an algorithm to extract the vibration-indicated data based on the point cloud, assuring low computational costs without losing vibration characteristics. The algorithm computes cuboids for two prominent clusters in every point cloud frame and sets motion points at the corners and centroid of the cuboid. The three-dimensional vector displacement of these points over consecutive point cloud frames, which corresponds to the vibration-affected clusters, are compiled as vibration indication data for each threshold class. A simply structured 1D Convolutional Neural Network (1D CNN)-based vibration threshold prediction model is proposed for fast, accurate, and real-time application. Finally, a threshold class mapping framework is developed which fuses the predicted threshold classes on the 3D occupancy map of the workspace, generating a 3D CbM map in real time, fostering a Condition-based Maintenance (CbM) strategy. The offline evaluation test results show an average accuracy of vibration threshold classes of 89.6% and consistent accuracy during real-time field case studies of 89%. The test outcomes validate that the proposed 3D-LiDAR-based CM framework is suitable for outdoor mobile robots, assuring the robot’s health and operational safety.

“…Hence, detecting the characteristics of vibration types using onboard sensors and predicting the sources of such anomalous vibrations using an easily executable DL technique is critical in mobile robots. Towards this, we introduced a vibration-based automated CM framework enabling CbM in previous works using an IMU sensor [18] and monocular camera sensor [19] individually. The camera-based study was better than the IMU-based study in real-time prediction time and accuracy.…”

Section: Problem Statementmentioning

confidence: 99%

AI-Enabled Condition Monitoring Framework for Indoor Mobile Cleaning Robots

Pookkuttath,

Veerajagadheswar,

Rajesh Elara

2023

Self Cite

Autonomous mobile cleaning robots are ubiquitous today and have a vast market need. Current studies are mainly focused on autonomous cleaning performances, and there exists a research gap on monitoring the robot’s health and safety. Vibration is a key indicator of system deterioration or external factors causing accelerated degradation or threats. Hence, this work proposes an artificial intelligence (AI)-enabled automated condition monitoring (CM) framework using two heterogeneous sensor datasets to predict the sources of anomalous vibration in mobile robots with high accuracy. This allows triggering proper maintenance or corrective actions based on the condition of the robot’s health or workspace, easing condition-based maintenance (CbM). Anomalous vibration sources are classified as induced by uneven Terrain, Collision with obstacles, loose Assembly, and unbalanced Structure, which causes accelerated system deterioration or potential hazards. Here, an unexplored heterogeneous sensor dataset using inertial measurement unit (IMU) and current sensors is proposed for effective recognition across different vibration classes, resulting in higher-accuracy prediction. A simple-structured 1D convolutional neural network (1D CNN) is developed for training and real-time prediction. A 2D CbM map is generated by fusing the predicted classes in real time on an occupancy grid map of the workspace to monitor the conditions of the robot and workspace remotely. The evaluation test results of the proposed method show that the usage of heterogeneous sensors performs significantly more accurately (98.4%) than previous studies, which used IMU (92.2%) and camera (93.8%) sensors individually. Also, this model is comparatively fast, fit for the environment, and ideal for real-time applications in mobile robots based on field trial validations, enhancing mobile robots’ productivity and operational safety.