Classifying BISINDO Alphabet using TensorFlow Object Detection API

Abstract

Indonesian Sign Language (BISINDO) is one of the sign languages used in Indonesia. The process of classifying BISINDO can be done by utilizing advances in computer technology such as deep learning. The use of the BISINDO letter classification system with the application of the MobileNet V2 FPNLite SSD model using the TensorFlow object detection API. The purpose of this study is to classify BISINDO letters A-Z and measure the accuracy, precision, recall, and cross-validation performance of the model. The dataset used was 4054 images with a size of consisting of 26 letter classes, which were taken by researchers by applying several research scenarios and limitations. The steps carried out are: dividing the ratio of the simulation dataset 80:20, and applying cross-validation (k-fold = 5). In this study, a real time testing using 2 scenarios was conducted, namely testing with bright light conditions of 500 lux and dim light of 50 lux with an average processing time of 30 frames per second (fps). With a simulation data set ratio of 80:20, 5 iterations were performed, the first iteration yielded a precision result of 0.758 and a recall result of 0.790, and the second iteration yielded a precision result of 0.635 and a recall result of 0.77, then obtained an accuracy score of 0.712, the third iteration provides a recall score of 0.746, the fourth iteration obtains a precision score of 0.713 and a recall score of 0.751, the fifth iteration gives a precision score of 0.742 for a fit score case and the recall score is 0.773. So, the overall average precision score is 0.712 and the overall average recall score is 0.747, indicating that the model built performs very well.

Deteksi Objek Bahasa Isyarat Alfabet BISINDO Menggunakan Deep Learning dan Arsitektur YOLO

“…Parameter accuracy, recall, dan precision yang didapatkan selama proses pengujian diketahui bahwa arsitektur YOLO memiliki kinerja yang baik. Penelitian sebelumnya [16] menggunakan arsitektur SSD Mobilenet V2 memiliki nilai accuracy sebesar 71,2%, nilai precision sebesar 71,2%, dan nilai recall sebesar 74,7%. Penelitian lain [15] yang menggunakan menggunakan arsitekur yang sama memiliki nilai accuracy sebesar 54,8% dan tidak melampirkan nilai recall dan precision.…”

“…Berbagai penelitian mengenai pemanfaatan pengolahan citra dan kecerdasan buatan untuk menyelesaikan permasalahan disabilitas tuna netra juga mengalami peningkatan. Penelitian mengenai bahasa isyarat menggunakan citra huruf BISINDO telah dilakukan sebelumnya menggunakan arsitektur SSD Mobilenet [15], [16]. Hasil dari penelitian tersebut menunjukkan bahwa bahasa isyarat BISINDO dapat dikenali menggunakan metode deep learning (DL) namun dapat lebih ditingkatkan kembali.…”

Hand image reading approach method to Indonesian Language Signing System (SIBI) using neural network and multi layer perseptron

“…The training program is a type of learning because the training method must be objective. It is called backward propagation because during training the error goes back to the base unit [24]. The network architecture used in this research is a multilayer network.…”