Fingertip interactive tracking registration method for AR assembly system

Abstract

Aiming at the problems of single input mode and lack of naturalness in the assembly process of existing AR systems, a tracking registration method of mobile AR assembly system is proposed based on multi-quantity and multi-degree of freedom natural fingertip interaction. Firstly, the real-time and stable tracking of hand area in complex environment is realized based on the hand region tracking; secondly, the fingertip detection and recognition based on K-COS and parallel vector is used to improve the precision and stability of fingertip recognition; thirdly, the special movement track of fingertip is recognized based on improved DTW algorithm, which has strong compatibility and feature gradient transformation for complex fingertip trajectory recognition; finally, through the real-time transformation of projection relationship between fingertip and virtual object, the interaction between fingertip and virtual object is made more natural and realistic. The experimental results show that in the complex environment of background, illumination, scale and rotation, the precision of fingertip detection and recognition is about 93%, and the precision of fingertip motion template matching is about 91%. The translation error of the registration method based on visual feature recognition is reduced by about 100pix compared with fingertip tracking registration method, and the efficiency of mobile AR-guided assembly method is improved by about 24.77% compared with the traditional manual assisted assembly method. These data verifies the strong interaction and practicability of the fingertips based on the user's multi-quantity and multi-degree of freedom features in the process of mobile AR guided assembly.

Exploring the effect of improved learning performance: A mobile augmented reality learning system

“…The MARLS included three major components: (1) interactive interfaces that include fingertip videos, interactive visual functions, and information rendering overlay; (2) hardware components that include camera, interactive semantics, and interactive data for introducing the hardware components (i.e., mainframe computer motherboard architecture, central processing unit, and random access memory (RAM)); (3) communication tools that include a fingertip interactive information acquisition module and an online registration function for the authentication of user identities (Jiu et al, 2022 ; Westerfield et al, 2015 ). In other words, the critical elements of the MARLS designed for this study include integrated real-world and virtual content using 3D AR-based technology and functions of real-time interaction (Belda-Medina & Calvo-Ferrer, 2022 ).…”

“…In this study, the participants can perform their learning tasks through 3D models and interactive omnidirectional videos that may increase their learning motivation and keep their cognitive resources available for learning (Skulmowski & Xu, 2022 ). A worth noting advantage of 3D-based learning functions, although they are more challenging for the human brains to process, is that they can overcome the disadvantages of the learning processes supported by two-dimensional (a flat object) visual presentation, such less effective and less interesting human–computer interactions and poor virtual control ability regarding the interactive assembly instructions (Alhonkoski et al, 2021 ; Jiu et al, 2022 ). Additionally, the main learning processes of the MARLS were performed by providing the learners with feedforward feedbacks for reducing unnecessary cognitive efforts devoted prior to the actual learning processes, and with cognitive feedbacks for enhancing learners’ odds of acquiring the accurate knowledge via a smart mobile device.…”

“…However, technologically enhanced learning (i.e., AR-based learning) is not always effective in terms of delivering knowledge to learners from the perspectives of information feedback and flow theory in different learning contexts (Cruz & Uresti, 2017 ; Guo & Ro, 2008 ). There is still a lot of ambiguity (e.g., interactive mode and degree of fidelity) in the findings of prior studies regarding the usefulness of MARLS, and there is a need for more studies on such applications (Maier et al, 2016 , Jiu et al, 2022 ; Skulmowski et al, 2022 ). Based on the literature review, the flow theory and cognitive load theory can help us disclose hidden information related to learners’ learning experience, including the improvement of their logical reasoning skills, the utilization of their cognitive resources, and the impact of information overload on the learners’ learning effectiveness (e.g., Chang et al, 2017 ; Hohnemann et al, 2022 ; Sun et al, 2019 ; Tang et al, 2022 ).…”