Learn2Write: Augmented Reality and Machine Learning-Based Mobile App to Learn Writing

Abstract

Augmented reality (AR) has been widely used in education, particularly for child education. This paper presents the design and implementation of a novel mobile app, Learn2Write, using machine learning techniques and augmented reality to teach alphabet writing. The app has two main features: (i) guided learning to teach users how to write the alphabet and (ii) on-screen and AR-based handwriting testing using machine learning. A learner needs to write on the mobile screen in on-screen testing, whereas AR-based testing allows one to evaluate writing on paper or a board in a real world environment. We implement a novel approach to use machine learning for AR-based testing to detect an alphabet written on a board or paper. It detects the handwritten alphabet using our developed machine learning model. After that, a 3D model of that alphabet appears on the screen with its pronunciation/sound. The key benefit of our approach is that it allows the learner to use a handwritten alphabet. As we have used marker-less augmented reality, it does not require a static image as a marker. The app was built with ARCore SDK for Unity. We further evaluated and quantified the performance of our app on multiple devices.

A Comprehensive Survey on the Investigation of Machine-Learning-Powered Augmented Reality Applications in Education

“…Among the models, EkushNet stood out due to its efficiency, achieving a test accuracy of 96.71%. The app not only assists with handwriting but also offers a promising avenue for enhancing early education through AR and ML [80].…”

“…Moreover, AR devices are underutilized in many fields, and both teachers and students may need training to maximize their potential. Additionally, implementing AR in educational settings may require additional resources and equipment [80]. ML models integrated into AR systems sometimes lack precision, hindering effective education.…”

A Comprehensive Survey on Investigation of Machine Learning-Powered Augmented Reality Applications in Education

“…Among the models, EkushNet stood out due to its efficiency, achieving a test accuracy of 96.71%. The app not only assists with handwriting but also offers a promising avenue for enhancing early education through AR and ML [77].…”

“…Moreover, AR devices are underutilized in many fields, and both teachers and students may need training to maximize their potential. Additionally, implementing AR in educational settings may require additional resources and equipment [77]. ML models integrated into AR systems sometimes lack precision, hindering effective education.…”

AI Educational Mobile App using Deep Learning Approach

“…The app has two features where AR-based is used to produce the alphabet on the screen, while machine learning works to detect the alphabet written. However, the appropriate learning environment is important, especially for the AR environment [17].…”