Detecting Hand-Head Occlusions in Sign Language Video

Abstract

A large body of current linguistic research on sign language is based on analyzing large corpora of video recordings. This requires either manual or automatic annotation of the videos. In this paper we introduce methods for automatically detecting and classifying hand-head occlusions in sign language videos. Linguistically, hand-head occlusions are an important and interesting subject of study as the head is a structural place of articulation in many signs. Our method combines easily calculable local video properties with more global hand tracking. The experiments carried out with videos of the Suvi on-line dictionary of Finnish Sign Language show that the sensitivity of the proposed local method in detecting occlusion events is 92.6%. When global hand tracking is combined in the method, the specificity can reach the level of 93.7% while still maintaining the detection sensitivity above 90%.

Distractor-Based Evaluation of Sign Spotting

“…To the best of our knowledge, few research contributions have proposed evaluation metrics for sign spotting. In [13], a tool to assess the spotting performance on continuous signing was developed. The evaluation is symmetrical, which is not considered ideal for retrieval metrics [24].…”

“…Like the sign spotting metric in [13], audio segmentation metrics and tolerances are used in a symmetrical manner. We distinguish our approach from them by proposing an asymmetrical evaluation.…”

“…Previous work in sign spotting has used a variety of tools, including dynamic time warping (e.g. [13,14]), conditional random fields (e.g. [15,16]) and hidden Markov models (e.g.…”

Exploring the Importance of Sign Language Phonology for a Deep Neural Network

“…We fine tune the last hidden layer of the model while keeping the rest of the layers frozen to ensure that all learning of sign language is embedded in the representations of this last layer. After exploration, we determine that the average length of signs in the CNGT is approximately 12 frames, corresponding to previous finds that co-articulated signs have length of approximately 7 to 13 frames [19,20,21]. Thus, we set the number of frames per input video to 16.…”

A Multi-Task Bayesian Deep Neural Net for Detecting Life-Threatening Infant Incidents From Head Images

“…We would like to develop a system that can assist any monitoring system in telling if an infant is in an improper sleeping position, causing her face being covered by the bedding. Existing works inferring occlusion on head images, such as [7,8,9], do not focus on analyzing infant images. On the contrary, we introduce a novel dataset named YunInfants and propose a method focusing on the infant group, aiming to handle any head pose and imaging under varying lighting conditions.…”