Environment Classification for Robotic Leg Prostheses and Exoskeletons Using Deep Convolutional Neural Networks

Abstract

Robotic leg prostheses and exoskeletons can provide powered locomotor assistance to older adults and/or persons with physical disabilities. However, the current locomotion mode recognition systems being developed for automated high-level control and decision-making rely on mechanical, inertial, and/or neuromuscular sensors, which inherently have limited prediction horizons (i.e., analogous to walking blindfolded). Inspired by the human vision-locomotor control system, we developed an environment classification system powered by computer vision and deep learning to predict the oncoming walking environments prior to physical interaction, therein allowing for more accurate and robust high-level control decisions. In this study, we first reviewed the development of our “ExoNet” database—the largest and most diverse open-source dataset of wearable camera images of indoor and outdoor real-world walking environments, which were annotated using a hierarchical labeling architecture. We then trained and tested over a dozen state-of-the-art deep convolutional neural networks (CNNs) on the ExoNet database for image classification and automatic feature engineering, including: EfficientNetB0, InceptionV3, MobileNet, MobileNetV2, VGG16, VGG19, Xception, ResNet50, ResNet101, ResNet152, DenseNet121, DenseNet169, and DenseNet201. Finally, we quantitatively compared the benchmarked CNN architectures and their environment classification predictions using an operational metric called “NetScore,” which balances the image classification accuracy with the computational and memory storage requirements (i.e., important for onboard real-time inference with mobile computing devices). Our comparative analyses showed that the EfficientNetB0 network achieves the highest test accuracy; VGG16 the fastest inference time; and MobileNetV2 the best NetScore, which can inform the optimal architecture design or selection depending on the desired performance. Overall, this study provides a large-scale benchmark and reference for next-generation environment classification systems for robotic leg prostheses and exoskeletons.

Stair Recognition for Robotic Exoskeleton Control using Computer Vision and Deep Learning

“…Compared to the classification results for the stair classes in the original ExoNet database [16], [20], [21], we achieved significantly higher prediction accuracies (Appendix 1). First, we discovered a number of ambiguous labelled images in the Ex-oNet classes LG-S, LG-T-DW, LG-T-IS, IS-S, IS-T-DW, and IS-T-LG.…”

“…This differentiation was not applied to our study such that the six ExoNet classes were combined into four. The final four environment classes in our dataset included: level ground (LG), which included both ExoNet classes “level ground steady state” and “level ground transition to door/wall”; level ground – incline stairs (LG-IS), which consisted of the ExoNet class “level ground transition to incline stairs”; incline stairs (IS), which included both ExoNet classes “incline stairs steady state” and “incline stairs transition to door/wall”; and incline stairs – level ground (LG-IS), which consisted of the ExoNet class “incline stairs transition to level ground.” The dataset was randomly split into training (89.5%), validation (3.5%), and testing (7%) sets, matching the subset distribution values from Laschowski and colleagues [16], while maintaining the class distributions– i.e., 85.8% for LG, 9.3% for IS, 3.1% for LG-IS, and 1.8% for IS-LG.…”

“…Andrew Garrett Kurbis, Brokoslaw Laschowski, and Alex Mihailidis state" and "incline stairs transition to door/wall"; and incline stairs -level ground (LG-IS), which consisted of the ExoNet class "incline stairs transition to level ground." The dataset was randomly split into training (89.5%), validation (3.5%), and testing (7%) sets, matching the subset distribution values from Laschowski and colleagues [16], while maintaining the class distributions-i.e., 85.8% for LG, 9.3% for IS, 3.1% for LG-IS, and 1.8% for IS-LG.…”

Improving IMU-Based Prediction of Lower Limb Kinematics in Natural Environments Using Egocentric Optical Flow

“…Convolutional layers have also been used to extract deeper features for robust performance in relatively complex environments [18], [19]. Recently, there has been greater interest in using deep learning methods for terrain and environment classification [20]- [23]. Then, environment features or classification decision can be fused with activity modes decision, using decision trees or SVM [12], [15].…”

“…Then, environment features or classification decision can be fused with activity modes decision, using decision trees or SVM [12], [15]. For a more exhaustive review of environment sensing techniques for terrain classification, refer to [1], [2], [23].…”