MobileASL: Intelligibility of sign language video over mobile phones

Abstract

For Deaf people, access to the mobile telephone network in the United States is currently limited to text messaging, forcing communication in English as opposed to American Sign Language (ASL), the preferred language. Because ASL is a visual language, mobile video phones have the potential to give Deaf people access to real-time mobile communication in their preferred language. However, even today's best video compression techniques can not yield intelligible ASL at limited cell phone network bandwidths. Motivated by this constraint, we conducted one focus group and two user studies with members of the Deaf Community to determine the intelligibility effects of video compression techniques that exploit the visual nature of sign language. Inspired by eye tracking results that show high resolution foveal vision is maintained around the face, we studied region-of-interest encodings (where the face is encoded at higher quality) as well as reduced frame rates (where fewer, better quality, frames are displayed every second). At all bit rates studied here, participants preferred moderate quality increases in the face region, sacrificing quality in other regions. They also preferred slightly lower frame rates because they yield better quality frames for a fixed bit rate. The limited processing power of cell phones is a serious concern because a real-time video encoder and decoder will be needed. Choosing less complex settings for the encoder can reduce encoding time, but will affect video quality. We studied the intelligibility effects of this tradeoff and found that we can significantly speed up encoding time without severely affecting intelligibility. These results show promise for real-time access to the current low-bandwidth cell phone network through sign-language-specific encoding techniques.

A web-based intelligibility evaluation of sign language video transmitted at low frame rates and bitrates

“…One would think that ASL, which is a temporal visual language, would require video communication to be transmitted at high frame rates; however, we discovered this is not the case at low bitrates. The preference of viewing ASL video at 10 fps over 15 fps was also discovered in earlier ASL video communication research conducted by Cavender et al [4] However, their findings only reported a slight but significant main effect that frame rate influenced video intelligibility. Our results strongly affirm that ASL video intelligibility peaks at 10 fps across all bitrates.…”

“…One fps was selected to achieve a sufficiently low frame rate to observe that intelligibility clearly suffered. Prior work investigating the impact of frame rate on perceived video quality acknowledged not selecting a low enough frame rate to explore in their study [4,16]. Although transmitting video at 1 fps is not ideal for ASL conversations, we did notice that transmitting video at 1 fps and 15 kbps, which is the lowest bitrate, received the highest mean Likert score across all bitrates at 1 fps.…”

“…Our research builds upon Cavender et al's [4] findings and more rigorously investigates intelligibility of sign language video. Cavender et al's laboratory study used prerecorded video filmed with a stationary video camera, which allowed more space in the signing region.…”

“…The researchers found that respondents favored video shown at 15 fps over 10 fps when shown at a fixed bitrate. Cavender et al [4] used ASL video clips and also discovered a frame rate preference for viewing ASL video at a fixed bitrate. They evaluated intelligibility of ASL video displayed at different frame rates (10 and 15 fps), bitrates (15,20, and 25 kbps), and region-of-interest (ROI) encoding levels (0, -6, and -12 ROI).…”

Analyzing the intelligibility of real-time mobile sign language video transmitted below recommended standards

“…Cavender et al [6] conducted a laboratory study evaluating perceived video intelligibility of pre-recorded ASL videos transmitted at two frame rates (10 and 15 fps), three bit rates (15,20, and 25 kbps), and three region-of-interest (ROI) encoding levels (0, -6, and -12 ROI). They discovered a frame rate preference of 10 fps for viewing ASL video at a fixed bit rate.…”