Handling Structural Mismatches in Real-time Opera Tracking

Abstract

Algorithms for reliable real-time score following in live opera promise a lot of useful applications such as automatic subtitles display, or real-time video cutting in live streaming. Until now, such systems were based on the strong assumption that an opera performance follows the structure of the score linearly. However, this is rarely the case in practice, because of different opera versions and directors' cutting choices. In this paper, we propose a two-level solution to this problem. We introduce a real-time-capable, high-resolution (HR) tracker that can handle jumps or repetitions at specific locations provided to it. We then combine this with an additional low-resolution (LR) tracker that can handle all sorts of mismatches that can occur at any time, with some imprecision, and can re-direct the HR tracker if the latter is 'lost' in the score. We show that the combination of the two improves tracking robustness in the presence of strong structural mismatches.

Wagner Ring Dataset: A Complex Opera Scenario for Music Processing and Computational Musicology

“…Concerning opera in general, a few datasets have been published on shorter works such as Mozart's Zauberflöte 5 or Weber's Freischütz. 6 MIR studies dealing with such earlier operas like Don Giovanni (Brazier and Widmer, 2021) or Freischütz (Röwenstrunk et al, 2015;Prätzlich and Müller, 2013) face challenges due to performance differences of the number operas, which lead to structural mismatches posing problems for music synchronization and tracking. Thanks to the throughcomposed style, the Ring is widely free of such structural problems but comes with other challenges.…”