Introspective Perception: Learning to Predict Failures in Vision Systems

Daftry, Shreyansh; Zeng, Sam; Bagnell, J. Andrew; Hebert, Martial

doi:10.48550/arxiv.1607.08665

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…That said, as these features deal with counts of operations, unknowns, and equations, a high degree of accuracy in creating the equations would not be required to faithfully generate such features. Following the ideas of machine learning introspection [9,10], we created performance prediction models using random forest and XGBoost. We utilized scikit-learn 1.0.2 and XGBoost 1.6.2 respectively.…”

Section: Performance Prediction Baselinesmentioning

confidence: 99%

“…This issue, combined with the ubiquity of such models has led to work on prompt generation and other aspects of the input [5,6]. Other areas of machine learning, such as meta-learning [7,8] and introspection [9,10] attempt to predict when a model will succeed or fail for a given input. An introspective tool, especially for certain tasks, could serve as a front-end to an LLM in a given application.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)

Shakarian¹,

Koyyalamudi²,

Ngu³

et al. 2023

Preprint

View full text Add to dashboard Cite

We study the performance of a commercially available large language model (LLM) known as ChatGPT on math word problems (MWPs) from the dataset DRAW-1K. To our knowledge, this is the first independent evaluation of ChatGPT. We found that ChatGPT's performance changes dramatically based on the requirement to show its work, failing 20% of the time when it provides work compared with 84% when it does not. Further several factors about MWPs relating to the number of unknowns and number of operations that lead to a higher probability of failure when compared with the prior, specifically noting (across all experiments) that the probability of failure increases linearly with the number of addition and subtraction operations. We also have released the dataset of ChatGPT's responses to the MWPs to support further work on the characterization of LLM performance and present baseline machine learning models to predict if ChatGPT can correctly answer an MWP. We have released a dataset comprised of ChatGPT's responses to support further research in this area.

show abstract

Section: Performance Prediction Baselinesmentioning

confidence: 99%