Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video

Moltisanti, Davide; Wray, Michael; Mayol-Cuevas, Walterio; Damen, Dima

doi:10.1109/iccv.2017.314

Cited by 28 publications

(26 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For each narrated sentence, we adjust the start and end times of the action using AMT. To ensure the annotators are trained to perform temporal localisation, we The green participant's annotations are selected as the final annotations use a clip from our previous work's understanding [33] that explains temporal bounds of actions. Each HIT is composed of a maximum of 10 consecutive narrated phrases p i , where annotators label A i = [t si , t ei ] as the start and end times of the i th action.…”

Section: Action Segment Annotationsmentioning

confidence: 99%

Scaling Egocentric Vision: The "Equation missing" Dataset

Damen

Doughty

Farinella

et al. 2018

Lecture Notes in Computer Science

Self Cite

562

942

View full text Add to dashboard Cite

First-person vision is gaining interest as it offers a unique viewpoint on people's interaction with objects, their attention, and even intention. However, progress in this challenging domain has been relatively slow due to the lack of sufficiently large datasets. In this paper, we introduce EPIC-KITCHENS, a large-scale egocentric video benchmark recorded by 32 participants in their native kitchen environments. Our videos depict non-scripted daily activities: we simply asked each participant to start recording every time they entered their kitchen. Recording took place in 4 cities (in North America and Europe) by participants belonging to 10 different nationalities, resulting in highly diverse cooking styles. Our dataset features 55 hours of video consisting of 11.5M frames, which we densely labelled for a total of 39.6K action segments and 454.3K object bounding boxes. Our annotation is unique in that we had the participants narrate their own videos (after recording), thus reflecting true intention, and we crowd-sourced ground-truths based on these. We describe our object, action and anticipation challenges, and evaluate several baselines over two test splits, seen and unseen kitchens.

show abstract

Section: Action Segment Annotationsmentioning

confidence: 99%

Scaling Egocentric Vision: The "Equation missing" Dataset

Damen

Doughty

Farinella

et al. 2018

Lecture Notes in Computer Science

Self Cite

562

942

View full text Add to dashboard Cite

show abstract

“…Fusion in Egocentric AR: Late fusion of appearance and motion has been frequently used in egocentric AR [8,24,38,40], as well as extended to additional streams aimed at capturing egocentric cues [21,37,38]. In [21], the spatial stream segments hands and detects objects.…”

Section: Related Workmentioning

confidence: 99%

EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition

Kazakos

Nagrani

Zisserman

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

300

202

View full text Add to dashboard Cite

We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multimodal temporal-binding, i.e. the combination of modalities within a range of temporal offsets. We train the architecture with three modalities -RGB, Flow and Audio -and combine them with mid-level fusion alongside sparse temporal sampling of fused representations. In contrast with previous works, modalities are fused before temporal aggregation, with shared modality and fusion weights over time. Our proposed architecture is trained end-to-end, outperforming individual modalities as well as late-fusion of modalities.We demonstrate the importance of audio in egocentric vision, on per-class basis, for identifying actions as well as interacting objects. Our method achieves state of the art results on both the seen and unseen test sets of the largest egocentric dataset: EPIC-Kitchens, on all metrics using the public leaderboard.

show abstract

“…Increasingly, datasets have been used for novel tasks, through pre-training (He et al 2019;Mettes et al 2016), self-supervision (Noroozi and Favaro 2016;Vondrick et al 2018) or additional annotations (Gupta and Malik 2016;Heilbron et al 2018). However, task adaptation demonstrates that models overfit to the data and annotations (Zhai et al 2019;Moltisanti et al 2017).…”

Section: Introduction and Related Datasetsmentioning

confidence: 99%

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

et al. 2021

Self Cite

View full text Add to dashboard Cite

This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (Damen in Scaling egocentric vision: ECCV, 2018), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the “test of time”—i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics.

show abstract

Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video

Cited by 28 publications

References 33 publications

Scaling Egocentric Vision: The "Equation missing" Dataset

Scaling Egocentric Vision: The "Equation missing" Dataset

EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

Contact Info

Product

Resources

About