Content-based tools for editing audio stories

Abstract

Audio stories are an engaging form of communication that combine speech and music into compelling narratives. Existing audio editing tools force story producers to manipulate speech and music tracks via tedious, low-level waveform editing. In contrast, we present a set of tools that analyze the audio content of the speech and music and thereby allow producers to work at much higher level. Our tools address several challenges in creating audio stories, including (1) navigating and editing speech, (2) selecting appropriate music for the score, and (3) editing the music to complement the speech. Key features include a transcript-based speech editing tool that automatically propagates edits in the transcript text to the corresponding speech track; a music browser that supports searching based on emotion, tempo, key, or timbral similarity to other songs; and music retargeting tools that make it easy to combine sections of music with the speech. We have used our tools to create audio stories from a variety of raw speech sources, including scripted narratives, interviews and political speeches. Informal feedback from first-time users suggests that our tools are easy to learn and greatly facilitate the process of editing raw footage into a final story.

Computer Vision – ECCV 2016 Workshops

“…Instead, we align the transcripts to speech audio, producing much greater timing precision and accuracy. Transcripts are force-aligned to speech-only audio for each episode using dynamic programming and an American-English language model, as in [24]. Alignment results in word and phoneme boundaries, as shown in Fig 2(a).…”

Generating emotionally relevant musical scores for audio stories

“…Rubin et al [22] focus on the process of editing speech and adding music to audio stories. Their system offers a highlevel editing workflow and requires a user-in-the-loop to refine speech and music edits.…”

“…Wenner et al [34] and Rubin et al [22] present methods for music retargeting to find optimal paths through music that match constraints of videos or audio stories. Our work builds on these algorithms and describes new constraints to match emotions of the speech and music, add pauses with flexible length, bound music segment lengths, and create scores from multiple music tracks.…”

“…We time-align the text transcript and the speech using a variant of the Penn Phonetics Lab Forced Aligner [36] from Rubin et al [22]. The timealignment allows our system to find the segment of speech that corresponds to each paragraph of the text.…”

“…To do this, our algorithm first finds the set of change point beats in the music using a novelty-based change point detection algorithm [8,22]. This algorithm computes a similarity matrix of per-beat music features, and then identifies large changes on the diagonal, which indicate significant differences in the music features.…”