The Effect of Computer-Generated Descriptions on Photo-Sharing Experiences of People with Visual Impairments

Abstract

Like sighted people, visually impaired people want to share photographs on social networking services, but find it difficult to identify and select photos from their albums. We aimed to address this problem by incorporating state-of-the-art computer-generated descriptions into Facebook's photo-sharing feature. We interviewed 12 visually impaired participants to understand their photo-sharing experiences and designed a photo description feature for the Facebook mobile application. We evaluated this feature with six participants in a seven-day diary study. We found that participants used the descriptions to recall and organize their photos, but they hesitated to upload photos without a sighted person's input. In addition to basic information about photo content, participants wanted to know more details about salient objects and people, and whether the photos reflected their personal aesthetic. We discuss these findings from the lens of self-disclosure and self-presentation theories and propose new computer vision research directions that will better support visual content sharing by visually impaired people.CCS Concepts: • Information interfaces and presentations → Multimedia and information systems; • Social and professional topics → People with disabilities Fig. 1. Computer-generated descriptions in Facebook. The text in the white boxes is the descriptions that are read to a blind user by TalkBack. The descriptions are normally invisible. We show it visually to demonstrate the design.While they want to share photos with others, visually impaired people find it difficult to understand the contents of photos and select good photos to post from their albums [30,47]. If a visually impaired person takes a photo and does not upload it immediately, it is hard for her to navigate through the album and find that photo independently, especially when many photos have accumulated in her album over time.Moreover, it is difficult to judge the quality of a photo, for example, to determine whether the photo is blurry, whether a person has her eyes closed in the photo, or whether the photo is aesthetically pleasing [30].Researchers and designers have tackled this problem, proposing techniques to help people with visual impairments access and understand the contents of a photo. Most of these efforts used human-powered services (e.g., crowd workers, friends) to provide photo descriptions or answer a photo-based question [13,15,54,55]. However, such systems are hard to scale and sustain due to the limited number of volunteers, the monetary cost of crowd workers, and the possible social costs involved when asking friends [18,22]. Also, since there can be private information in local photos without the user knowing it, human-powered services also exposed visually impaired users to high privacy risks when sending personal photos to human assistants [16]. Therefore, other approaches are necessary for captioning a number of personal photos from a user's album in a photo-sharing use case.Recent research by Wu et al. presented automatic al...

The Care Work of Access

“…The design and use of ATs is a now established thread of research in HCI. Relevant, for example, are projects that have used computer vision to support people with vision impairments to complete tasks like identifying objects, people, and the contents of photos on social media [8], [40], [41], [49], [53], [78], [88], [90], [91], [92], [93], [94], [95]. For example, VizWiz [8] allowed blind people to photograph images for algorithms or crowd workers to describe.…”

“…For example, VizWiz [8] allowed blind people to photograph images for algorithms or crowd workers to describe. Subsequently, a suite of apps and services providing such access have become widely available and affordable [4], [27], [55], [88], [91]. However, these apps and services respond little to social environments and cues.…”

“…In the data driven approaches popular in contemporary AI systems and being introduced into ATs [4], [8], [27], [40], [41], [49], [53], [58], [78], [88], [90], [91], [92], [93], [94], [95] these mundane attunements present a challenge. When what matters in the work of access is open, attuned in response to so many contingencies, how do we find datasets that might afford aggregation in any meaningful sense and that lead to the identification of actionable patterns of interaction that aren't too general?…”

"I Hope This Is Helpful"

“…While some definitions focus specifically on creating "good" captions for people who are blind [9,11,22,79,84,94,95,100,108], only a few studies directly integrate preferences reported by people who are blind [94]. To our knowledge, our work is the first to identify crowdworkers' questions and concerns about how to create good image captions-especially for people who are blind.…”

Image Accessibility for Screen Reader Users: A Systematic Review And A Road Map

“…For instance, images of maps were investigated 169 the most (N = 10), followed by graphs (N = 6). Interestingly, while the accessibility of 170 photographs for BLV were largely investigated in terms of web accessibility [20,38], only 171 three out of 33 papers aimed to support photographs particularly using touchscreen 172 devices. In addition, as touchscreen devices themselves have accessibility issues for 173 people with visual impairments requiring accurate hand-eye coordination [25], four 174 papers focused on improving the accessibility of the touchscreen-based interface itself 175 such as soft buttons [39][40][41] and gestures [42].…”