Temporal Action Detection with Structured Segment Networks

Zhao, Yue; Xiong, Yuanjun; Wang, Limin; Wu, Zhirong; Tang, Xiaoou; Lin, Dahua

doi:10.1109/iccv.2017.317

Cited by 871 publications

(655 citation statements)

References 53 publications

Supporting

Mentioning

646

Contrasting

Unclassified

Order By: Relevance

“…As a task we pay more attention to, the step localization is related to the area of action detection, where promising progress has also been achieved recently [39], [40], [42], [43]. For example, Zhao et al [80] developed structured segment networks (SSN) to model the temporal structure of each action instance with a structured temporal pyramid. Xu et al [74] introduced a Region Convolutional 3D Network (R-C3D) architecture, which was built on C3D [71] and Faster R-CNN [51], to explore the region information of video frames.…”

Section: Replace a Bulb Install A Ceiling Fanmentioning

confidence: 99%

“…Bottom-up Aggregation: As our method is built upon the proposal-based action detection methods, we start with training an existing action detector, e.g., SSN [80], on our COIN dataset. During inference phase, given an input video, we send it into the action detector to produce a series of proposals with their corresponding locations and predicted scores.…”

Section: Task-consistency Analysismentioning

confidence: 99%

“…Then, at time-stamp t, we sum the scores of all the steps to calculate its corresponding binary actionness score a(t). After that, we follow [80] to apply the watershed algorithm on the 1D signal a(t) to obtain a sequence of segments S = {t l s , s l s } L l=1 , where its validness is theoretically guaranteed by [55]. Here, t l s = [t l1 s , t l2 s ] is the temporal interval of the segment S l , where we calculate the step score as:…”

Section: Ordering-dependency Analysismentioning

confidence: 99%

“…We uniformly segmented the video into three intervals, and randomly assigned the label to each interval. (2) SSN [80]. This is an effective model for action detection, which outputs the same type of results (interval and label for each action 5.…”

Section: Evaluation On Step Localizationmentioning

confidence: 99%

“…Following the default setting, we set split 1 as testing set and the other splits as training set. Similar to COIN, we employ SSN [80], which is a state-of-the-art method for action detection, as a baseline method under the setting of step localization. As shown in Table 9, our proposed task-consistency and ordering-dependency methods improve the performance of the baseline model, which further shows their advantages to model the dependencies of different steps in instructional videos.…”

Section: Evaluation On Step Localizationmentioning

confidence: 99%

See 4 more Smart Citations

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Thanks to the substantial and explosively inscreased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, the most existing datasets in this area have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, we present a large-scale dataset named as "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on the COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source code are available at http://coin-dataset.github.io. DomainTaskStep VehiclesHousehold Items Change the Car Tire {unscrew the screws, jack up the car, remove the tire, put on the tire, tighten the screws } {remove the door knob, remove bolt and pin board, install new pin board, install new bolt, install new door knob } Replace the Door Knob

show abstract

Section: Replace a Bulb Install A Ceiling Fanmentioning

confidence: 99%

Section: Task-consistency Analysismentioning

confidence: 99%

Section: Ordering-dependency Analysismentioning

confidence: 99%

Section: Evaluation On Step Localizationmentioning

confidence: 99%

Section: Evaluation On Step Localizationmentioning

confidence: 99%

See 3 more Smart Citations

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

VAL: Visual-Attention Action Localizer

Song

Han

2018

Advances in Multimedia Information Processing – PCM 2018

View full text Add to dashboard Cite

Audio-Visual Event Localization in Unconstrained Videos

Tian

Jing

et al. 2018

Lecture Notes in Computer Science

364

381

View full text Add to dashboard Cite

In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos. We define an audio-visual event as an event that is both visible and audible in a video segment. We collect an Audio-Visual Event (AVE) dataset to systemically investigate three temporal localization tasks: supervised and weakly-supervised audio-visual event localization, and cross-modality localization. We develop an audio-guided visual attention mechanism to explore audio-visual correlations, propose a dual multimodal residual network (DMRN) to fuse information over the two modalities, and introduce an audio-visual distance learning network to handle the cross-modality localization. Our experiments support the following findings: joint modeling of auditory and visual modalities outperforms independent modeling, the learned attention can capture semantics of sounding objects, temporal alignment is important for audio-visual fusion, the proposed DMRN is effective in fusing audio-visual features, and strong correlations between the two modalities enable cross-modality localization.achieves the best fusion results. For weakly-supervised learning, we formulate it as a Multiple Instance Learning (MIL) [11] task, and modify our network structure via adding a MIL pooling layer to handle the problem. To address the harder cross-modality localization task, we propose an audio-visual distance learning network that measures the relativeness of any given pair of audio and visual content. It projects audio and visual features into subspaces with the same dimension. Contrastive loss [12] is introduced to learn the network.Observing that there is no publicly available dataset directly suitable for our tasks, we collect a large video dataset that consists of 4143 10-second videos with both audio and video tracks for 28 audio-visual events and annotate their temporal boundaries. Videos in our dataset are originated from YouTube, thus they are unconstrained. Our extensive experiments support the following findings: modeling jointly over auditory and visual modalities outperforms modeling independently over them, audio-visual event localization in a noisy condition can still achieve promising results, the audio-guided visual attention can well capture semantic regions covering sounding objects and can even distinguish audio-visual unrelated videos, temporal alignment is important for audio-visual fusion, the proposed dual multimodal residual network is effective in addressing the fusion task, and strong correlations between the two modalities enable crossmodality localization. These findings have paved a way for our community to solve harder, high-level understanding problems in the future, such as video captioning [13] and movieQA [14], where the auditory modality plays an important role in understanding video but lacks effective modeling.Our work makes the following contributions: (1) a family of three audio-visual event localization tasks; (2) an audio-guided visual attention model to adaptively explore the audio-visual corre...

show abstract

Temporal Action Detection with Structured Segment Networks

Cited by 871 publications

References 53 publications

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

VAL: Visual-Attention Action Localizer

Audio-Visual Event Localization in Unconstrained Videos

Contact Info

Product

Resources

About