Temporal Recurrent Networks for Online Action Detection

Xu, Mingze; Gao, Mingfei; Chen, Yi-Ting; Davis, Larry S.; Crandall, David J.

doi:10.1109/iccv.2019.00563

Cited by 159 publications

(220 citation statements)

References 55 publications

Supporting

Mentioning

219

Contrasting

Order By: Relevance

“…Temporal Action Localization in long videos is widely studied in both offline and online scenarios. In the offline setting, temporal action detectors (Shou et al, 2016;Buch et al, 2017;Gao et al, 2017c;Chao et al, 2018) predict the start and end times of actions after observing the whole video, while online approaches (De Geest et al, 2016;Gao et al, 2017b;Shou et al, 2018b;Xu et al, 2018;Gao et al, 2019) label action class in a per-frame manner without accessing future information. The goal of temporal action detectors is to localize actions in pre-defined categories.…”

Section: Related Workmentioning

confidence: 99%

WSLLN:Weakly Supervised Natural Language Localization Networks

Gao¹,

Davis²,

Socher³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

View full text Add to dashboard Cite

We propose weakly supervised language localization networks (WSLLN) to detect events in long, untrimmed videos given language queries. To learn the correspondence between visual segments and texts, most previous methods require temporal coordinates (start and end times) of events for training, which leads to high costs of annotation. WSLLN relieves the annotation burden by training with only video-sentence pairs without accessing to temporal locations of events. With a simple end-to-end structure, WSLLN measures segment-text consistency and conducts segment selection (conditioned on the text) simultaneously. Results from both are merged and optimized as a video-sentence matching problem. Experiments on ActivityNet Captions and DiDeMo demonstrate that WSLLN achieves state-of-the-art performance.

show abstract

Section: Related Workmentioning

confidence: 99%

WSLLN:Weakly Supervised Natural Language Localization Networks

Gao¹,

Davis²,

Socher³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

Self Cite

View full text Add to dashboard Cite

show abstract

“…Gao et al [12] propose a Reinforced Encoder-Decoder network for action anticipation and treat online action detection as a special case of their framework. Temporal Recurrent Networks [38] set a new state-of-the-art performance by conducting current and future action detection jointly. With the same goal of online per-frame labeling, these methods can serve as ClsNet in our framework.…”

Section: Related Workmentioning

confidence: 99%

“…Inspired by recent online action detection methods [9,12,38], we utilize recurrent networks, specifically, LSTM [19], to construct ClsNet. At each time t, it uses the previous hid-…”

Section: Classification Network (Clsnet)mentioning

confidence: 99%

“…Temporal action localization (TAL) in untrimmed videos has been widely studied in offline settings, where start and end times of an action are recognized after the action is fully observed [4,7,8,13,30,39]. With the emerging applications that require identifying actions in real time, e.g., autonomous driving, surveillance system, and collaborative robots, online action detection (OAD) methods [9,12,29,38] have been proposed. They typically pose the TAL problem as a per-frame class labeling task.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

StartNet: Online Detection of Action Start in Untrimmed Videos

Gao

Davis

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

View full text Add to dashboard Cite

We propose StartNet to address Online Detection of Action Start (ODAS) where action starts and their associated categories are detected in untrimmed, streaming videos. Previous methods aim to localize action starts by learning feature representations that can directly separate the start point from its preceding background. It is challenging due to the subtle appearance difference near the action starts and the lack of training data. Instead, StartNet decomposes ODAS into two stages: action classification (using ClsNet) and start point localization (using LocNet). ClsNet focuses on per-frame labeling and predicts action score distributions online. Based on the predicted action scores of the past and current frames, LocNet conducts class-agnostic start detection by optimizing long-term localization rewards using policy gradient methods. The proposed framework is validated on two large-scale datasets, THUMOS'14 and ActivityNet. The experimental results show that StartNet significantly outperforms the state-of-the-art by 15%-30% p-mAP under the offset tolerance of 1-10 seconds on THU-MOS'14, and achieves comparable performance on Activi-tyNet with ×10 smaller time offset. * Work done when the author was an intern at Salesforce Research. † Corresponding author. Action Score (b) StartNet Unseen Action Start AS Action Classification Start Point Localization t No Yes (a)

show abstract

“…Thus, our method fuses the information from both parts. Due to the good performance of recurrent networks [26], [27], [9] on online action detection tasks, our framework is based on LSTM [15].…”

Section: Model Descriptionmentioning

confidence: 99%

Goal-oriented Object Importance Estimation in On-road Driving Videos

Gao

Tawari

Martin

2019

2019 International Conference on Robotics and Automation (ICRA)

Self Cite

View full text Add to dashboard Cite

We formulate a new problem as Object Importance Estimation (OIE) in on-road driving videos, where the road users are considered as important objects if they have influence on the control decision of the ego-vehicle's driver. The importance of a road user depends on both its visual dynamics, e.g., appearance, motion and location, in the driving scene and the driving goal, e.g., the planned path, of the ego vehicle. We propose a novel framework that incorporates both visual model and goal representation to conduct OIE. To evaluate our framework, we collect an on-road driving dataset at traffic intersections in the real world and conduct human-labeled annotation of the important objects. Experimental results show that our goal-oriented method outperforms baselines and has much more improvement on the left-turn and right-turn scenarios. Furthermore, we explore the possibility of using object importance for driving control prediction and demonstrate that binary brake prediction can be improved with the information of object importance. * Work done during an internship at the Honda Research Institute, USA.

show abstract

Temporal Recurrent Networks for Online Action Detection

Cited by 159 publications

References 55 publications

WSLLN:Weakly Supervised Natural Language Localization Networks

WSLLN:Weakly Supervised Natural Language Localization Networks

StartNet: Online Detection of Action Start in Untrimmed Videos

Goal-oriented Object Importance Estimation in On-road Driving Videos

Contact Info

Product

Resources

About