Combined audio and video analysis for guitar chord identification

Abstract

This paper presents a multi-modal approach to automatically identifying guitar chords using audio and video of the performer. Chord identification is typically performed by analyzing the audio, using a chroma based feature to extract pitch class information, then identifying the chord with the appropriate label. Even if this method proves perfectly accurate, stringed instruments add extra ambiguity as a single chord or melody may be played in different positions on the fretboard. Preserving this information is important, because it signifies the original fingering, and implied "easiest" way to perform the selection. This chord identification system combines analysis of audio to determine the general chord scale (i.e. A major, G minor), and video of the guitarist to determine chord voicing (i.e. open, barred, inversion), to accurately identify the guitar chord.

Multimodal Music Information Processing and Retrieval: Survey and Future Challenges

“…• music segmentation, in which audio and video, lyrics or note level can be exploited to identify the music piece structure [56]- [58]; • spatial transcription, that is the inference, starting from audio and video, of the note level of songs for fretted instruments, so that the resulting score includes the annotation of fingering [59], [60]; • onset detection through audio and performer video [61] or rhythmic structure knowledge;…”