Essential Features in a Theory of Context for Enabling Artificial General Intelligence

Kejriwal, Mayank

doi:10.3390/app112411991

Cited by 5 publications

(2 citation statements)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ultimately, such language representation models would be used in the real world, not only in multiple‐choice settings, but also in so‐called generative settings where the model may be expected to generate answers to questions (without being given options). Even in the multiple‐choice setting, without robust commonsense, the model will likely not be usable for actual decision making unless we can trust that it is capable of generalization (Kejriwal, 2021; Misra, 2022; Wahle et al, 2022). One option to implementing such robustness in practice may be to add a ‘decision‐making layer’ on a pre‐trained language representation model rather than aim to modify the model's architecture from scratch 13 (Hong et al, 2021; Tang & Kejriwal, 2022; Zaib et al, 2020).…”

Section: Discussionmentioning

confidence: 99%

An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

Shen

Kejriwal

2023

Expert Systems

Self Cite

View full text Add to dashboard Cite

In the last 5 years, language representation models, such as BERT and GPT-3, based on transformer neural networks, have led to enormous progress in natural language processing (NLP). One such NLP task is commonsense reasoning, where performance is usually evaluated through multiple-choice question answering benchmarks. Till date, many such benchmarks have been proposed, and 'leaderboards' tracking stateof-the-art performance on those benchmarks suggest that transformer-based models are approaching human-like performance. Because these are commonsense benchmarks, however, such a model should be expected to generalize, that is, at least in aggregate, should not exhibit excessive performance loss across independent commonsense benchmarks regardless of the specific benchmark on (the training set of) which it has been fine-tuned. In this article, we evaluate this expectation by proposing a methodology and experimental study to measure the generalization ability of language representation models using a rigorous and intuitive metric. Using five established commonsense reasoning benchmarks, our experimental study shows that the models do not generalize well, and may be (potentially) susceptible to issues such as dataset bias. The results therefore suggest that current performance on benchmarks may be an over-estimate, especially if we want to use such models on novel commonsense problems for which a 'training' dataset may not be available, for the language representation model, to fine-tune on.

show abstract

Section: Discussionmentioning

confidence: 99%

An experimental study measuring the generalization of fine‐tuned language representation models across commonsense reasoning benchmarks

Shen

Kejriwal

2023

Expert Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…To conclude the issue, Kejriwal [5] offers a perspective on how context will prove critical to building robust artificial general intelligence (AGI) architectures. In particular, I argue that context needs to become a focal point of the conceptual landscape, rather than a vague topic of discussion in AI papers.…”

mentioning

confidence: 99%