Towards Aircraft Maintenance Metaverse Using Speech Interactions with Virtual Objects in Mixed Reality

Abstract

Metaverses embedded in our lives create virtual experiences inside of the physical world. Moving towards metaverses in aircraft maintenance, mixed reality (MR) creates enormous opportunities for the interaction with virtual airplanes (digital twin) that deliver a near-real experience, keeping physical distancing during pandemics. 3D twins of modern machines exported to MR can be easily manipulated, shared, and updated, which creates colossal benefits for aviation colleges who still exploit retired models for practicing. Therefore, we propose mixed reality education and training of aircraft maintenance for Boeing 737 in smart glasses, enhanced with a deep learning speech interaction module for trainee engineers to control virtual assets and workflow using speech commands, enabling them to operate with both hands. With the use of the convolutional neural network (CNN) architecture for audio features and learning and classification parts for commands and language identification, the speech module handles intermixed requests in English and Korean languages, giving corresponding feedback. Evaluation with test data showed high accuracy of prediction, having on average 95.7% and 99.6% on the F1-Score metric for command and language prediction, respectively. The proposed speech interaction module in the aircraft maintenance metaverse further improved education and training, giving intuitive and efficient control over the operation, enhancing interaction with virtual objects in mixed reality.

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

“…The convergence of virtual augmented physical reality and physically permanent virtual space Siyaev and Jo [52] Stunning mixed reality digital space inside the physical world, interacting in millions of 3D virtual experiences With the requirement of physical distancing, an online digital environment that allows people to share an increasingly diverse human experience.…”

“…Maharg and Owen [204] conducted a study on application simulation for educational purposes. Siyaev and Jo [52] conducted a study on virtual assets and workflow control using aircraft engineer voice commands. In the case of a virtual environment based on the real environment, exaggeration and the intention of the creator can be included in the process of describing the environment in Metaverse.…”

“…In addition to simply focusing on interest, there are ways to approach to simplify difficult tasks through games. As much as payment and personal information are widely used in Metaverse, a game based on blockchain technology has been proposed [52]. Hide and Seek is a simple yet effective simulation environment for multi-agent work that uses visual representations of objects and scenes from an egocentric perspective [210][211][212][213].…”

De la Realidad Extendida al Metaverso: una reflexión crítica sobre las aportaciones a la educación

“…La brecha digital en países y escuelas (Maas & Hughes, 2020). Las experiencias de RM en el metaverso requerirán de interacción verbal con objetos virtuales y recibir un feedback o retroalimentación, de modo que la interacción de voz es el reto principal para pasar de la RM al desarrollo del metaverso en la educación (Siyaev & Jo, 2021). Para ello, es preciso perfeccionar el desarrollo de las gafas empleadas en la RM, que funcionan con comandos de voz estáticos y predefinidos (Eschen et al, 2018), e incluir otros flexibles que se adapten al feedback de los usuarios.…”

A Survey of the State of Research on Augmented Reality from a Business Perspective using Porter’s Value Chain

“…As AR in itself is a technology on its way to general adaptation, the field of technical progress is also to be related to AR itself, as the ever emerging applications also bring opportunities to cooperate with, bring forth or further develop other innovative technologies [44]- [46]. Especially innovative technologies such as virtual reality (VR), blockchain technologies, as for example in intelligent mobile augmented reality (IMAR) projects, or a future metaverse benefit from the development of AR, as intersections keep emerging, which is ultimately extremely interesting for companies, on the one hand financially, on the other hand image-wise [45], [47], [48].…”