Feature Combination Meets Attention: Baidu Soccer Embeddings and Transformer based Temporal Detection

Abstract

With rapidly evolving internet technologies and emerging tools, sports related videos generated online are increasing at an unprecedentedly fast pace. To automate sports video editing/highlight generation process, a key task is to precisely recognize and locate the events in the long untrimmed videos. In this tech report, we present a two-stage paradigm to detect what and when events happen in soccer broadcast videos. Specifically, we fine-tune multiple action recognition models on soccer data to extract high-level semantic features, and design a transformer based temporal detection module to locate the target events. This approach achieved the state-of-the-art performance in both two tasks, i.e., action spotting and replay grounding, in the SoccerNet-v2 Challenge, under CVPR 2021 ActivityNet workshop. Our soccer embedding features are released at https://github.com/baidu-research/vidpress-sports. By sharing these features with the broader community, we hope to accelerate the research into soccer video understanding.* denotes equal contributions. The order was determined using Python's random.shuffle()

SoccerNet 2022 Challenges Results

“…The baseline performance correspond to last year's winner [32]. As shown in Table 2, this year's winning method significantly improved the spotting performance for tight intervals in both the challenge and test sets.…”

Spotting Temporally Precise, Fine-Grained Events in Video

“…In Table 4, we compare E2E-Spot to the best results from the CVPR 2021 (lenient tolerances of 5-60 sec) and CVPR 2022 (less coarse, 1-5 sec tolerances) SoccerNet Action Spotting challenges [14]. E2E-Spot, with the 200MF CNN, matches the top prior method from the 2021 competition [75] in the 5-60 sec setting while outperforming it by 13.7-14.1 avg-mAP points in the less coarse, 1-5 sec setting. Increasing the CNN to 800MF improves avg-mAP slightly (by 0.4-2.7 avg-mAP).…”

“…E2E-Spot places second in the (concurrent) 2022 competition (within 1.1 avg-mAP), after Soares et al [54], due to the latter's strong performance on unshown actions (not visible in the frame). Soares et al [54,55] and Zhou et al [75] are two-phase approaches, combining pre-extracted features from multiple (5 to 6) heterogeneous, fine-tuned feature extractors and proposing downstream architectures and losses on those features. In contrast, E2E-Spot shows that direct, end-to-end training of a simple and compact model can be a surprisingly strong baseline.…”

“…The baselines use the following head architectures: MS-TCN [20], GRU [8], ASFormer [68] from TAS; GCN [66] from TAD; and NetVLAD++ [23] and transformer [75] from action spotting. MS-TCN, GRU, and ASFormer performed best in our experiments, so we relegate results from the remaining architectures to § C.1.…”

“…StridedTransformer. We implement a transformer [45] that operates on a window of per-frame features [75]. The model takes a consecutive clip of 31 features and positional encodings, and it predicts whether the center frame is one of the K events or not.…”