An Adaptive Thresholding-Based Movement Epenthesis Detection Technique Using Hybrid Feature Set for Continuous Fingerspelling Recognition

Abstract

Sign language recognition systems are gaining importance in recent times as these have established themselves as important elements of human-computer interaction. Also these provide an opportunity for the deaf and hearing impaired to communicate with the common people without the need of an interpreter. Yet there are plenty of challenges in this field which are worth exploring and solutions formulated. In this paper, we have addressed the design of a continuous fingerspelling recognition system which segments a fingerspelling sequence into meaningful extracts and non-sign patterns and thereby recognizes the meaningful signs. Sign segmentation is carried out by means of a unique set of features which comprises of shape matching, velocity change and displacement of centroid between successive frames. Specialized techniques like adaptive thresholding and finite state machine model are also incorporated into our system for efficient classification of sign and movement epenthesis frames. We have validated the performance of our proposed system taking into account the continuous fingerspelling alphabets of American Sign Language considering both native and non-native signers and have obtained an accuracy of almost 91.29%. Our proposed system also has the potential to tackle complex backgrounds involving multiple objects, backgrounds with multiple signers and different brightness conditions.

Analytical review of models and methods for automatic recognition of gestures and sign languages

“…Основным отличием данной задачи от задачи распознавания статических жестов является наличие временных признаков и зачастую отсутствие предварительной разметки (т. е. ассоциации последовательностей кадров с жестами). Другую проблему, связанную с распознаванием слитной речи, представляет так называемая эпентеза -межжестовые движения, возникающие в потоке речи [44] В работе [51] представлена гибридная архитектура CNN для распознавания жестов в формате RGB-D, нейросетевая модель, которая извлекает пространственно-временные характеристики двух модальностей с использованием комбинации 3D CNN и LSTM. Временная информация входных данных модели кодируется с использованием двухпоточной архитектуры, основанной на уменьшенной модели сети ResNet-10.…”