Learning Goals from Failure

Epstein, Dave; Vondrick, Carl

doi:10.1109/cvpr46437.2021.01104

Cited by 13 publications

(11 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…dataset of unintentional actions, which mostly includes human actions. Models for automated detection of action intention have been developed from this dataset Epstein and Vondrick [42]. These and other examples are discussed in detail in Section 5.…”

Section: What Is Human Error?mentioning

confidence: 99%

“…Compared to a baseline model only trained on the Kinetics action recognition dataset [26], their model performed similarly when classifying intentional vs. unintentional actions when only using video speed as a predictive feature. Epstein and Vondrick [42] later built on this work by adding annotations to each of the videos with a decoder. Their 3D-CNN model outperformed a Kinetics-trained model.…”

Section: Unsupervised Learningmentioning

confidence: 99%

See 1 more Smart Citation

Using Social Cues to Recognize Task Failures for HRI: A Review of Current Research and Future Directions

Bremers¹,

Pabst²,

Parreira³

et al. 2023

Preprint

View full text Add to dashboard Cite

Robots that carry out tasks and interact in complex environments will inevitably commit errors. Error detection is thus an important ability for robots to master, to work in an efficient and productive way. People leverage social cues from others around them to recognize and repair their own mistakes. With advances in computing and AI, it is increasingly possible for robots to achieve a similar error detection capability. In this work, we review current literature around the topic of how social cues can be used to recognize task failures for human-robot interaction (HRI). This literature review unites insights from behavioral science, human-robot interaction, and machine learning, to focus on three areas: 1) social cues for error detection (from behavioral science), 2) recognizing task failures in robots (from HRI), and 3) approaches for autonomous detection of HRI task failures based on social cues (from machine learning). We propose a taxonomy of error detection based on self-awareness and social feedback. Finally, we leave recommendations for HRI researchers and practitioners interested in developing robots that detect (physical) task errors using social cues from bystanders.

show abstract

Section: What Is Human Error?mentioning

confidence: 99%

Section: Unsupervised Learningmentioning

confidence: 99%

Using Social Cues to Recognize Task Failures for HRI: A Review of Current Research and Future Directions

Bremers¹,

Pabst²,

Parreira³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Previous works on unintentional action prediction or anomaly detection [11, 14, 28-30, 35, 40] do not address representation learning, but focus on predictions based on features pre-extracted from pre-trained networks. Epstein et al [9,10] instead proposed to learn features specifically for the tasks related to unintentional action prediction. For the unsupervised learning setting, Epstein et al [9] proposed three baselines: Video Speed, Video Sorting and Video Context.…”

Section: Self-supervised Learning For Unintentional Action Recognitionmentioning

confidence: 99%

“…For the unsupervised learning setting, Epstein et al [9] proposed three baselines: Video Speed, Video Sorting and Video Context. In the further work, Epstein et al [10] have also considered fully-supervised learning, for which they combined learning on labeled examples using the standard cross-entropy loss with solving an unsupervised temporal consistency task. Two further works [37,41] addressed the fully-supervised setting for the Oops dataset.…”

Section: Self-supervised Learning For Unintentional Action Recognitionmentioning

confidence: 99%

Self-supervised Learning for Unintentional Action Prediction

Zatsarynna

Farha

Gall

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Distinguishing if an action is performed as intended or if an intended action fails is an important skill that not only humans have, but that is also important for intelligent systems that operate in human environments. Recognizing if an action is unintentional or anticipating if an action will fail, however, is not straight-forward due to lack of annotated data. While videos of unintentional or failed actions can be found in the Internet in abundance, high annotation costs are a major bottleneck for learning networks for these tasks. In this work, we thus study the problem of self-supervised representation learning for unintentional action prediction. While previous works learn the representation based on a local temporal neighborhood, we show that the global context of a video is needed to learn a good representation for the three downstream tasks: unintentional action classification, localization and anticipation. In the supplementary material, we show that the learned representation can be used for detecting anomalies in videos as well.

show abstract

“…Our work is, to some degree, relevant to future prediction -a popular research area in computer vision. In this area, a huge spectrum of topics/tasks are put forward, including forecasting future frames [42,70], future features [62,69], future actions [1,28,31,55,61], future human motions [14,22,41], future human trajectories [2], future goals [13], etc. Rather than studying the future generation at the semantic-category or color-pixel level, event-level prediction was recently addressed in [34] and [49].…”

Section: Related Workmentioning

confidence: 99%

Visual Abductive Reasoning

Chen¹,

Yang²,

Zhou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Abductive reasoning seeks the likeliest possible explanation for partial observations. Although abduction is frequently employed in human daily reasoning, it is rarely explored in computer vision literature. In this paper, we propose a new task and dataset, Visual Abductive Reasoning (VAR), for examining abductive reasoning ability of machine intelligence in everyday visual situations. Given an incomplete set of visual events, AI systems are required to not only describe what is observed, but also infer the hypothesis that can best explain the visual premise. Based on our large-scale VAR dataset, we devise a strong baseline model, REASONER (causal-and-cascaded reasoning Transformer). First, to capture the causal structure of the observations, a contextualized directional position embedding strategy is adopted in the encoder, that yields discriminative representations for the premise and hypothesis. Then, multiple decoders are cascaded to generate and progressively refine the premise and hypothesis sentences. The prediction scores of the sentences are used to guide cross-sentence information flow in the cascaded reasoning procedure. Our VAR benchmarking results show that REASONER surpasses many famous video-language models, while still being far behind human performance. This work is expected to foster future efforts in the reasoning-beyond-observation paradigm.* Part of this work was done when Chen Liang was an intern at Baidu.

show abstract

Learning Goals from Failure

Cited by 13 publications

References 31 publications

Using Social Cues to Recognize Task Failures for HRI: A Review of Current Research and Future Directions

Using Social Cues to Recognize Task Failures for HRI: A Review of Current Research and Future Directions

Self-supervised Learning for Unintentional Action Prediction

Visual Abductive Reasoning

Contact Info

Product

Resources

About