Recurrent Neural Network for Recognition of Gestures of the Russian Language, Taking into Account the Language Dialect of the Siberian Region

Abstract

Sign recognition is an important task, in particular for the communication of the deaf and hard of hearing population with people who do not know sign language. Russian sign language is poorly studied, Russian sign language of the Siberian region has significant differences from others within the Russian language group. There is no generally accepted data set for Russian Sign Language. The paper presents a gesture recognition algorithm based on video data. The gesture recognition algorithm is based on the identification of key features of the hands and posture of a person. Gestures were classified using the LSTM recurrent neural network. To train and test the results of gesture recognition, we independently developed a data set consisting of 10 sign words. The selection of words for the data set was made among the most popular words of the Russian language, as well as taking into account the maximum difference in the pronunciation of gestures of the language dialect of the Siberian region. The implementation of the gesture recognition algorithm was carried out using Keras neural network design and deep learning technologies, the OpenCV computer vision library, the MediaPipe machine learning framework, and other auxiliary libraries. Experimental studies conducted on 300 video sequences confirm the effectiveness of the proposed algorithm.