AI-Powered Smart Glasses for Sensing and Recognition of Human-Robot Walking Environments

Rossos, Daniel; Mihailidis, Alex; Laschowski, Brokoslaw

doi:10.1101/2023.10.24.563804

Cited by 7 publications

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks

Ivanyuk-Skulskiy,

Kurbis,

Mihailidis

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Robotic prosthetic legs and exoskeletons require real-time and accurate estimation of the walking environment for smooth transitions between different locomotion mode controllers. However, previous studies have mainly been limited to static image classification, therein ignoring the temporal dynamics of human-robot locomotion. Motivated by these limitations, here we developed several state-of-the-art temporal convolutional neural networks (CNNs) to compare the performances between static vs. sequential image classification of real-world walking environments (i.e., level-ground terrain, incline stairs, and transitions to and from stairs). Using our large-scale image dataset, we trained a number of encoder networks such as VGG, MobileNetV2, ViT, and MobileViT, each coupled with a temporal long short-term memory (LSTM) backbone. We also trained MoViNet, a new video classification model designed for mobile and embedded devices, to further compare the performances between 2D and 3D temporal deep learning models. Our 3D network outperformed all the hybrid 2D encoders with LSTM backbones and the 2D CNN baseline model in terms of classification accuracy, suggesting that network architecture can play an important role in performance. However, although our 3D neural network achieved the highest classification accuracy, it had disproportionally higher computational and memory storage requirements, which can be disadvantageous for real-time control of robotic leg prostheses and exoskeletons with limited onboard resources.

show abstract

Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks

Ivanyuk-Skulskiy,

Kurbis,

Mihailidis

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Egocentric Perception of Walking Environments using an Interactive Vision-Language System

Tan,

Mihailidis,

Laschowski

2024

Preprint

View full text Add to dashboard Cite

Large language models can provide a more detailed contextual understanding of a scene beyond what computer vision alone can provide, which have implications for robotics and embodied intelligence. In this study, we developed a novel multimodal vision-language system for egocentric visual perception, with an initial focus on real-world walking environments. We trained a number of state-of-the-art transformer-based vision-language models that use causal language modelling on our custom dataset of 43,055 image-text pairs for few-shot image captioning. We then designed a new speech synthesis model and a user interface to convert the generated image captions into speech for audio feedback to users. Our system also uniquely allows for feedforward user prompts to personalize the generated image captions. Our system is able to generate detailed captions with an average length of 10 words while achieving a high ROUGE-L score of 43.9% and a low word error rate of 28.1% with an end-to-end processing time of 2.2 seconds. Overall, our new multimodal vision-language system can generate accurate and detailed descriptions of natural scenes, which can be further augmented by user prompts. This innovative feature allows our image captions to be personalized to the individual and immediate needs and preferences of the user, thus optimizing the closed-loop interactions between the human and generative AI models for understanding and navigating of real-world environments.

show abstract

Comparative Analysis of Neural Decoding Algorithms for Brain-Machine Interfaces

Shevchenko,

Yeremeieva,

Laschowski

2024

Preprint

View full text Add to dashboard Cite

Accurate neural decoding of brain dynamics remains a significant and open challenge in brain-machine interfaces. While various signal processing, feature extraction, and classification algorithms have been proposed, a systematic comparison of these is lacking. Accordingly, here we conducted one of the largest comparative studies evaluating different combinations of state-of-the-art algorithms for motor neural decoding to find the optimal combination. We studied three signal processing methods (i.e., artifact subspace reconstruction, surface Laplacian filtering, and data normalization), four feature extractors (i.e., common spatial patterns, independent component analysis, short-time Fourier transform, and no feature extraction), and four machine learning classifiers (i.e., support vector machine, linear discriminant analysis, convolutional neural networks, and long short-term memory networks). Using a large-scale EEG dataset, we optimized each combination for individual subjects (i.e., resulting in 672 total experiments) and evaluated performance based on classification accuracy. We also compared the computational and memory storage requirements, which are important for real-time embedded computing. Our comparative analysis provides novel insights that help inform the design of next-generation neural decoding algorithms for brain-machine interfaces used to interact with and control robots and computers.

show abstract

AI-Powered Smart Glasses for Sensing and Recognition of Human-Robot Walking Environments

Cited by 7 publications

References 21 publications

Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks

Sequential Image Classification of Human-Robot Walking Environments using Temporal Neural Networks

Egocentric Perception of Walking Environments using an Interactive Vision-Language System

Comparative Analysis of Neural Decoding Algorithms for Brain-Machine Interfaces

Contact Info

Product

Resources

About