Static and Dynamic Concepts for Self-supervised Video Representation Learning

Fine-Grained Spatiotemporal Motion Alignment for Contrastive Video Representation Learning

“…In linear probing settings, our method achieves the best results on both datasets. As the major counterparts with R(2+1)D backbone, DCLR [15] and SDC [43] also utilize frame difference as the source of motion information. FIMA outperforms DCLR and SDC in general, which implies we align the features of frame difference more precisely.…”

“…In the video domain, this learning diagram has also presented promising performance by keeping the instances within the same video semantically consistent [16,45]. However, vanilla video contrastive learning has difficulty modeling local temporal information [12,43] and possesses severe background bias [14,55], which limits the generalization and transferability of the learned representations. The reason may stem from the existence of static bias in the positive pair construction.…”

“…But its quality is extremely susceptible to background noise caused by camera jitter or drastic changes in the background. Several approaches [15,43,49,50] employ it in video contrastive learning and show promising results. Even with notable improvements in performance, how to effectively align the RGB and frame difference features in the latent space still not be fully explored.…”

The exact asymptotic form of Bayesian generalization error in latent Dirichlet allocation

“…For instance, following [28], when we predict the knee arthritis grades of patients by using x-ray images and a CBM, we set the concepts as clinical findings corrected by medical doctors, and thereby understand how clinical findings affect the predicted grades based on the correlation, by observing the learned weights in the last connection. Concept-based interpretation is used in knowledge discovery for chess [39], video representation [44], medical imaging [25], clinical risk prediction [45], computer aided diagnosis [27], and other healthcare domain problems [10]. CBM is a significant foundation for these applications, and advanced methods [49,44,27] have been proposed based on CBM.…”

“…Concept-based interpretation is used in knowledge discovery for chess [39], video representation [44], medical imaging [25], clinical risk prediction [45], computer aided diagnosis [27], and other healthcare domain problems [10]. CBM is a significant foundation for these applications, and advanced methods [49,44,27] have been proposed based on CBM. Hence, it is important to clarify the theoretical behavior of CBM.…”