Taking a Cue From the Human

Starr, Kim; Braun, Sabine; Delfani, Jaleh

doi:10.47476/jat.v3i2.2020.138

Cited by 5 publications

(8 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Results of the present study could, for instance, make audio describers aware of the importance of verbalising event changes in relation to the spatiotemporal context and offer examples and solutions as to how this can be done. Event segmentation plays also an important role for automated, computer-generated video description (Braun et al, 2020;Starr et al, 2020). It is necessary to teach the algorithms to identify actions in dynamic scenes, to "see" connections between frames and actions, and to recognise event boundaries.…”

Section: Discussionmentioning

confidence: 99%

“…Event segmentation is also important for computer-generated video description, a new promising area that is being rapidly developed to supplement human audio description and make audio-visual media more widely accessible. However, as the comparisons of human and automatic scene descriptions show, there are fundamental problems with the current state and quality of machinegenerated descriptions (Braun et al, 2020;Starr et al, 2020). Among other issues, computer algorithms "see" images in isolation as single frames, do not integrate them, and are likely to miss several key actions (or mis-label them).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Event Segmentation in the Audio Description of Films

Holsanova,

Blomberg,

Blomberg

et al. 2023

JAT

View full text Add to dashboard Cite

To make the content of films available to a visually impaired audience, a sighted translator can provide audio description (AD), a verbal description of visual events. To achieve this goal, the audio describer needs to select what to describe, when to describe it, and how to describe it, as well as to express the information aurally. The efficacy of this communication is critically dependent upon basic cognitive processes of how the sighted audio describer perceives and segments the film’s unfolding chain of events and in what way the visually impaired end users conceive the structure, content, and segmentation of such events in relation to the produced AD. There is, however, virtually no research on this interplay in relation to AD. In this study, we scrutinize live AD of a film from two trained audio describers and examine how events are structured, segmented and construed in their AD. Results demonstrate that the event segmentation experienced from the film is indeed a fundamental part of how AD is structured and construed. It was found that AD at event boundaries was highly sensitive to different spatiotemporal circumstances and this relationship depends on semantic resources for expressing AD. Lay summary In our everyday lives, we naturally organize and remember our experiences as meaningful sequences of events. This happens the same way when we watch movies – we see them as a series of events happening in different places, times, and with changing character dynamics. Now, think about making this cinematic experience accessible to people with visual impairments. This is where a sighted translator, called an audio describer, becomes really important. To make movies understandable for a visually impaired audience, a sighted audio describer describes crucial visual events in the film. Their goal is to create a "narrative equivalence" between the original film and the audio-described version. Thus, the main challenges for audio describers are deciding what information to describe, when to do it, and how to articulate it effectively. In the current case study, we looked at two audio descriptions for the Swedish film "Skumtimmen" (English title: "Echoes from the Dead"). These descriptions were created by trained audio describers. We wanted to see how these narrators organized, broke down, and verbalized the course of events in their audio descriptions. To achieve this, we focused on two important parts of storytelling – changes in time and location – and paid special attention to how often and in what way audio describers verbalized these event boundaries. What we found is that both describers indeed verbalized most event boundaries. This shows that these boundaries are a big part of how audio descriptions are put together. We also learned that the descriptions at these event boundaries were very sensitive to when and where key events unfolded in the story. The way these boundaries were described depended on fundamental cognitive and linguistic resources to talk about space and time. Our findings help us understand how breaking down events in a film is a crucial part of shaping audio descriptions. Plus, our research highlights how important it is to study how events are broken down and verbalized, giving useful insights to improve audio descriptions and train future audio describers. The ultimate goal is to make the experience better for visually impaired viewers, contributing to a more inclusive and enjoyable audiovisual world.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Event Segmentation in the Audio Description of Films

Holsanova,

Blomberg,

Blomberg

et al. 2023

JAT

View full text Add to dashboard Cite

show abstract

“…The process of event segmentation is equally essential for automated computer-generated video description (Braun, Starr, & Laaksonen, 2020;Starr, Braun, & Delfani, 2020;Starr & Braun, 2023). content on an equal footing with their sighted counterparts.…”

Section: Event Boundary Perception In Ad Filmsmentioning

confidence: 99%

“…Consequently, a central inquiry in the context of successful AD pertains to how the sequence of events unfolding in a film aligns with the verbal narration conveyed through AD. Event segmentation also holds significance in the domain of computer-generated video description, an emerging field aimed at augmenting human-generated AD to make audio-visual media more widely accessible (Braun, Starr & Laaksonen, 2020;Starr, Braun & Delfani, 2020).…”

mentioning

confidence: 99%

Event Boundary Perception in Audio Described Films by People Without Sight

Johansson,

Rastegar,

Åhlander

et al. 2023

Preprint

View full text Add to dashboard Cite

Audio description (AD) serves a critical role in making film narratives accessible to visually impaired audiences, aiming to enhance their viewing experience and comprehension. One method to assess the comprehension of film narratives, is through an event segmentation task, wherein participants delineate the narrative unfolding into distinct meaningful events. In the present study, both sighted and visually impaired participants engaged in such tasks. Sighted participants watched a Swedish film, while visually impaired participants experienced the same film with two AD versions—one explicitly expressing key event boundaries and another containing more implicitly conveyed ones. Our findings indicate that visually impaired participants perceived event boundaries similarly to sighted participants, suggesting that AD effectively conveys the event structure. However, in the AD version with implicit expressions, event boundaries were less likely to be recognized. These results shed light on event segmentation dynamics in films, emphasizing the importance of how event boundaries are presented in AD. This has significant implications for improving the cinematic experience for visually impaired viewers, emphasizing the need for clear, explicit information about event boundaries within AD.

show abstract

“…In the European MeMAD project (grant no. 780069), our primary focus has been on developing semi-automated video description models which replicate, as far as possible, the work of human describers of audiovisual content [1,2]. This has been achieved using computer vision modelling, theories of human engagement with multimodal narrative, and the integration of machine-generated data within an editing platform, Flow, which draws together machine descriptions, named-entity recognition, metadata, transcriptions and translation services (Fig.…”

Section: Study Aims and Structurementioning

confidence: 99%

When Worlds Collide: AI-Created, Human-Mediated Video Description Services and the User Experience

Braun

Starr

Delfani

et al. 2021

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

This paper reports on a user-experience study undertaken as part of the H2020 project MeMAD ('Methods for Managing Audiovisual Data: Combining Automatic Efficiency with Human Accuracy'), in which multimedia content describers from the television and archive industries tested Flow, an online platform, designed to assist the post-editing of automatically generated data, in order to enhance the production of archival descriptions of film content. Our study captured the participant experience using screen recordings, the User Experience Questionnaire (UEQ), a benchmarked interactive media questionnaire and focus group discussions, reporting a broadly positive post-editing environment. Users designated the platform's role in the collation of machine-generated content descriptions, transcripts, named-entities (location, persons, organisations) and translated text as helpful and likely to enhance creative outputs in the longer term. Suggestions for improving the platform included the addition of specialist vocabulary functionality, shot-type detection, film-topic labelling, and automatic music recognition. The limitations of the study are, most notably, the current level of accuracy achieved in computer vision outputs (i.e. automated video descriptions of film material) which has been hindered by the lack of reliable and accurate training data, and the need for a more narratively oriented interface which allows describers to develop their storytelling techniques and build descriptions which fit within a platform-hosted storyboarding functionality. While this work has value in its own right, it can also be regarded as paving the way for the future (semi)automation of audio descriptions to assist audiences experiencing sight impairment, cognitive accessibility difficulties or for whom 'visionless' multimedia consumption is their preferred option.

show abstract

Taking a Cue From the Human

Cited by 5 publications

References 32 publications

Event Segmentation in the Audio Description of Films

Event Segmentation in the Audio Description of Films

Event Boundary Perception in Audio Described Films by People Without Sight

When Worlds Collide: AI-Created, Human-Mediated Video Description Services and the User Experience

Contact Info

Product

Resources

About