Stair Recognition for Robotic Exoskeleton Control using Computer Vision and Deep Learning

Abstract

Computer vision can be used in robotic exoskeleton control to improve transitions between different locomotion modes through the prediction of future environmental states. Here we present the development of a large-scale automated stair recognition system powered by convolutional neural networks to recognize indoor and outdoor real-world stair environments. Building on the ExoNet database – the largest and most diverse open-source dataset of wearable camera images of walking environments – we designed a new computer vision dataset, called StairNet, specifically for stair recognition with over 515,000 images. We then developed and optimized an efficient deep learning model for automatic feature engineering and image classification. Our system was able to accurately predict complex stair environments with 98.4% classification accuracy. These promising results present an opportunity to increase the autonomy and safety of human-exoskeleton locomotion for real-world community mobility. Future work will explore the mobile deployment of our automated stair recognition system for onboard real-time inference.

Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks

“…To train and test our image classification models, we used the open-source StairNet dataset [21] with four environment classes: level-ground terrain (LG), level-ground transition to incline stairs (LG-IS), incline stairs (IS), and incline stairs transition to level-ground (IS-LG). The StairNet dataset includes videos recorded in urban environments using a wearable camera.…”

“…Recent studies [12]-[19] have focused on using deep learning and large-scale datasets, such as ExoNet [20] and StairNet [21], to develop systems that can generalize to diverse walking environments. These systems use convolutional neural networks (CNNs) and transfer learning for image classification such that the model weights are trained on large datasets like ImageNet [22] and fine-tuned on downstream tasks with mod-…”

“…Cameras have been used for environment state estimation in combination with inertial sensors, used for locomotor state estimation [2]. Early systems for environment recognition were limited to statistical pattern recognition and machine learning algorithms that required manual feature engineering and/or were developed using relatively small image datasets [3]- [11].Recent studies [12]-[19] have focused on using deep learning and large-scale datasets, such as ExoNet [20] and StairNet [21], to develop systems that can generalize to diverse walking environments. These systems use convolutional neural networks (CNNs) and transfer learning for image classification such that the model weights are trained on large datasets like ImageNet [22] and fine-tuned on downstream tasks with mod-…”

Efficient Visual Perception of Human-Robot Walking Environments using Semi-Supervised Learning

“…However, the dataset has many environmental classes that overlap with one another with varying degrees, resulting in poor generalization. Kurbis and colleagues [8]- [9] recently developed a 4-class image dataset called StairNet based on ExoNet by combining similar classes and focusing on stair recognition. The system allowed for more consistent and robust training of a convolutional neural network.…”

“…Our goal is to support the development of new computer vision systems for environment-adaptive control of robotic prosthetic legs and exoskeletons. This work focuses on improving training efficiency and making computer vision systems more accessible to researchers in wearable robotics by minimizing the number of required labelled images while maintaining high prediction accuracy similar to the previous state-of-the-art [8]- [9].…”