Prediction and Description of Near-Future Activities in Video

Mahmud, Tahmida; Billah, Mohammad; Hasan, Mahmudul; Roy-Chowdhury, Amit K.

doi:10.48550/arxiv.1908.00943

Search citation statements

Order By: Relevance

Paper Sections

Select...

Action Anticipation1

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2021

Publication Types

Select...

Other1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, we are the first to predict the future in the form of sentences instead of category-based predictions as in [16], [49], [50]. Similar to us, the recent work [51] also predicts sentences for subsequent actions by extending their anticipation framework [41].…”

Section: Action Anticipationmentioning

confidence: 75%

Transferring Knowledge from Text to Video: Zero-Shot Anticipation for Procedural Actions

Sener¹,

Saraf²,

Yao³

2021

Preprint

View full text Add to dashboard Cite

Can we teach a robot to recognize and make predictions for activities that it has never seen before? We tackle this problem by learning models for video from text. This paper presents a hierarchical model that generalizes instructional knowledge from large-scale text-corpora and transfers the knowledge to video. Given a portion of an instructional video, our model recognizes and predicts coherent and plausible actions multiple steps into the future, all in rich natural language. To demonstrate the capabilities of our model, we introduce the Tasty Videos Dataset V2, a collection of 4022 recipes for zero-shot learning, recognition and anticipation. Extensive experiments with various evaluation metrics demonstrate the potential of our method for generalization, given limited video data for training models.

show abstract