Real-time recognition framework for Indian Sign Language using fine-tuned convolutional neural networks

Abstract

Sign language is a very crucial aspect in the lives of those who cannot speak or listen to those around them. People with these disabilities have difficulty communicating with the outside world and feel left behind. Much research is ongoing to create a better way of communicating for these people. This work establishes interaction between hearing or speech impaired with the world by recognizing the 33-hand pose and gestures of Indian Sign Language (ISL). This framework can recognize alphabets and numbers in real-time and also generate gestures in real-time for the given alphabets and numbers. The fine-tuned Convolutional Neural Network (CNN) model is explored for the recognition of alphabets and numbers in real-time. A GUI is developed for an easy-to-use interface and immediate visual feedback. Data acquisition software is also developed to create a database. A database of 74,200 images of 33 static signs is captured and used in this work. The results are evaluated on different CNN architectures and learning rates. Accuracy, precision, recall, and F-score are used as performance metrics. The proposed work accomplished the most noteworthy training precision of 99.97% and a validation accuracy of 99.59%.