Automatic Synchronization between Lyrics and Music CD Recordings Based on Viterbi Alignment of Segregated Vocal Signals

Abstract

This paper describes a system that can automatically synchronize between polyphonic musical audio signals and corresponding lyrics. Although there were methods that can synchronize between monophonic speech signals and corresponding text transcriptions by using Viterbi alignment techniques, they cannot be applied to vocals in CD recordings because accompaniment sounds often overlap with vocals. To align lyrics with such vocals, we therefore developed three methods: a method for segregating vocals from polyphonic sound mixtures, a method for detecting vocal sections, and a method for adapting a speech-recognizer phone model to segregated vocal signals. Experimental results for 10 Japanese popular-music songs showed that our system can synchronize between music and lyrics with satisfactory accuracy for 8 songs.

Integrating Additional Chord Information Into HMM-Based Lyrics-to-Audio Alignment

“…The baseline method [7] is based on an HMM in which each phoneme is represented by three hidden states, and the observed nodes correspond to the low-level feature, which we will call phoneme feature. Given a phoneme state, the 25 elements of the phoneme feature vector consist of 12 MFCCs, 12 MFCCs and 1 element containing the power difference (the subscript m stands for MFCC).…”

“…More audio-centric approaches aimed at word-level alignment employ a hidden Markov model (HMM) and forced alignment [2], [7], [22]: Chen et al [2] use a VAD component to restrict alignment to vocal areas. Mesaros and Virtanen's HMM [22] use audio features based on the singing voice automatically segregated from the audio, but little attention is devoted to VAD: verse or chorus sections are manually selected.…”

“…Mesaros and Virtanen's HMM [22] use audio features based on the singing voice automatically segregated from the audio, but little attention is devoted to VAD: verse or chorus sections are manually selected. Both vocal activity detection and singing voice segregation are addressed in [7], where a left-to-right HMM architecture is used to align lyrics to audio, based on observed Mel frequency cepstral coefficients (MFCCs). We have chosen this more complete approach as a baseline method for our research (see Section III).…”

A Tandem Algorithm for Singing Pitch Extraction and Voice Separation From Music Accompaniment

“…We thus calculate the ACF using the envelopes of filtered signals. The corresponding cross-channel correlation is calculated similarly to (7). Here, we extract envelopes by halfwave rectification and bandpass filtering [11].…”

“…They use matched filtering to model pitch likelihoods and a Gaussian mixture model (GMM) to model pitch transitions [21]. In a model-based method [7], sinusoidal models are used to model harmonic partials given detected pitch. The models are used to create smooth amplitude and phase trajectories over time and sinusoids are generated and summed to produce an estimate of the vocal signal.…”