Action Coherence Network for Weakly-Supervised Temporal Action Localization

Zhai, Yuanhao; Wang, Le; Tang, Wei; Zhang, Qilin; Zheng, Nanning; Hua, Gang

doi:10.1109/tmm.2021.3073235

Cited by 23 publications

(9 citation statements)

References 71 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Multiple methods use features extracted from a pre-trained two-stream model, I3D [32], as input to their weakly supervised model [23][24][25]. In addition to MIL, supervision on feature similarity or difference between clips in videos [23,25,33] and adversarial erasing of clip predictions [34,35] are also used to encourage localization predictions that are temporally complete. In early stages of our study, we mainly attempted a MIL approach, but the noisy data, low number of samples and similar appearance of videos from the two classes were prohibitive for such a model to learn informative patterns.…”

Section: Weakly Supervised Action Recognition and Localizationmentioning

confidence: 99%

Sharing pain: Using pain domain transfer for video recognition of low grade orthopedic pain in horses

et al. 2022

View full text Add to dashboard Cite

Orthopedic disorders are common among horses, often leading to euthanasia, which often could have been avoided with earlier detection. These conditions often create varying degrees of subtle long-term pain. It is challenging to train a visual pain recognition method with video data depicting such pain, since the resulting pain behavior also is subtle, sparsely appearing, and varying, making it challenging for even an expert human labeller to provide accurate ground-truth for the data. We show that a model trained solely on a dataset of horses with acute experimental pain (where labeling is less ambiguous) can aid recognition of the more subtle displays of orthopedic pain. Moreover, we present a human expert baseline for the problem, as well as an extensive empirical study of various domain transfer methods and of what is detected by the pain recognition method trained on clean experimental pain in the orthopedic dataset. Finally, this is accompanied with a discussion around the challenges posed by real-world animal behavior datasets and how best practices can be established for similar fine-grained action recognition tasks. Our code is available at https://github.com/sofiabroome/painface-recognition.

show abstract

Section: Weakly Supervised Action Recognition and Localizationmentioning

confidence: 99%

Sharing pain: Using pain domain transfer for video recognition of low grade orthopedic pain in horses

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Multiple methods use features extracted from a pre-trained twostream model, I3D [8], as input to their weakly supervised model [20,29,33]. In addition to MIL, supervision on feature similarity or difference between clips in videos [20,29,48], and adversarial erasing of clip predictions [36,47] are also used to encourage localization predictions that are temporally complete.…”

Section: Related Workmentioning

confidence: 99%

Sharing Pain: Using Pain Domain Transfer for Video Recognition of Low Grade Orthopedic Pain in Horses

Broomé,

Ask,

Rashid

et al. 2021

Preprint

View full text Add to dashboard Cite

Orthopedic disorders are a common cause for euthanasia among horses, which often could have been avoided with earlier detection. These conditions often create varying degrees of subtle but long-term pain. It is challenging to train a visual pain recognition method with video data depicting such pain, since the resulting pain behavior also is subtle, sparsely appearing, and varying, making it challenging for even an expert human labeler to provide accurate ground-truth for the data. We show that transferring features from a dataset of horses with acute nociceptive pain (where labeling is less ambiguous) can aid the learning to recognize more complex orthopedic pain. Moreover, we present a human expert baseline for the problem, as well as an extensive empirical study of various domain transfer methods and of what is detected by the pain recognition method trained on acute pain in the orthopedic dataset. Finally, this is accompanied with a discussion around the challenges posed by real-world animal behavior datasets and how best practices can be established for similar fine-grained action recognition tasks. Our code is available at https://github.com/sofiabroome/ painface-recognition.

show abstract

“…Set-supervised Learning. The set of actions present in training videos is assumed known in [9,21,22,23,25,32,34,35,36,40,41,45,43,44,7]. For example, Shou et al [32] specified the outer-inner-contrastive loss for learning an action boundary detector, Nguyen et al [23] defined a background-aware loss to distinguish actions from the background, and Paul et al [25] proposed an action affinity loss for multi-instance learning.…”

Section: Related Workmentioning

confidence: 99%

Set-Constrained Viterbi for Set-Supervised Action Segmentation

Todorovic

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

This paper is about action segmentation under weak supervision in training, where the ground truth provides only a set of actions present, but neither their temporal ordering nor when they occur in a training video. We use a Hidden Markov Model (HMM) grounded on a multilayer perceptron (MLP) to label video frames, and thus generate a pseudo-ground truth for the subsequent pseudo-supervised training. In testing, a Monte Carlo sampling of action sets seen in training is used to generate candidate temporal sequences of actions, and select the maximum posterior sequence. Our key contribution is a new anchor-constrained Viterbi algorithm (ACV) for generating the pseudo-ground truth, where anchors are salient action parts estimated for each action from a given ground-truth set. Our evaluation on the tasks of action segmentation and alignment on the benchmark Breakfast, MPII Cooking2, Hollywood Extended datasets demonstrates our superior performance relative to that of prior work.

show abstract

Action Coherence Network for Weakly-Supervised Temporal Action Localization

Cited by 23 publications

References 71 publications

Sharing pain: Using pain domain transfer for video recognition of low grade orthopedic pain in horses

Sharing pain: Using pain domain transfer for video recognition of low grade orthopedic pain in horses

Sharing Pain: Using Pain Domain Transfer for Video Recognition of Low Grade Orthopedic Pain in Horses

Set-Constrained Viterbi for Set-Supervised Action Segmentation

Contact Info

Product

Resources

About