Automated Video Description for Blind and Low Vision Users

“It’s Kind of Context Dependent”: Understanding Blind and Low Vision People’s Video Accessibility Preferences Across Viewing Scenarios

“…• Level of Detail: Prior work has investigated the potential for visual question answering systems to enable users to query for details that they wish to know [6,58,110]. As AI advances, it may one day be possible to provide end users with high degrees of flexibility for which details and what level of detail they would like through automatically generated descriptions.…”

“…Other research has shown that BLV people wish to interact and engage with video content in ways other than only listening to preset neutral descriptions during the video itself. For example, Stangl et al [110] and Bodi et al [6] investigated the viability of providing video access through interactive visual question answering, reinforcing the importance of BLV users having agency in the process of making videos accessible. Others explored the impact of changing the tone or style of verbal descriptions for select video types, finding that alternative AD styles were engaging for BLV users [30,59,121,125].…”

“…Some research has focused on the development of datasets and NLP techniques for video understanding and accessibility [42,47,48,115,131,138]. Others have developed AI-based tools to support accessibility practices [6,11,66,93,110,126,134,135]. Major advancements in multi-modal language models such as OpenAI's GPT-4V [90, 91] and Google's Gemini [24,116] show that AI is already capable of generating image descriptions, and some video descriptions, that attain high levels of quality [135] and BLV user satisfaction [23,110].…”

“…To support video accessibility, researchers and practitioners have aimed to increase the quantity of described videos through crowdsourcing platforms and automation (e.g., [6,11,61,84,126,132]). Others have focused on improving the quality of AD through providing authorship guidelines to help determine what content to include or which tone of voice to use (e.g., [3, 26-28, 58, 59, 117]).…”

NarrationBot and InfoBot: A Hybrid System for Automated Video Description

“…Automated approaches to video description, currently dominated by deep learning, are usually divided into two stages: 1) visual content extraction or the encoding stage and 2) text generation or the decoding stage. For encoding, convolutional neural networks (CNNs) [71] are used to learn visual features, and for decoding, different variations of recurrent neural networks (RNNs), such as long short-term memory (LSTM) [37] and gated recurrent unit (GRU) [19] networks are used for language modeling and text generation [14,40]. Recent state-of-the-art methods [47,69] have replaced the RNNs with BERT [25] due to the success of Transformers [72].…”