The ZuCo Benchmark on Cross-Subject Reading Task Classification with EEG and Eye-Tracking Data

Hollenstein, Nora; Tröndle, Marius; Płomecka, Martyna Beata; Kiegeland, Samuel; Özyurt, Yilmazcan; Jäger, Lena A.; Langer, Nicolas

doi:10.1101/2022.03.08.483414

Cited by 4 publications

(2 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to reading and language comprehension, eye-tracking has also been used in understanding how humans perform other linguistic tasks such as annotation [18,19]. Previous studies [20,21,22] have analyzed eye tracking data such as number of xations, search time, and xation duration recorded during manual annotation of named-entities in texts to a) estimate the di culty level based on cognitive load assessed using gaze data eye-tracking data, and, b) identify prominent features used by humans for named-entity recognition.…”

Section: Related Work On Eye Trackingmentioning

confidence: 99%

Comparing Human Text Classification Performance and Explainability with Large Language and Machine Learning Models Using Eye-Tracking

Nanda,

Venkatesh,

Jaiswal

2024

Preprint

View full text Add to dashboard Cite

To understand the alignment between reasonings of humans and artificial intelligence (AI) models, this empirical study compared the human text classification performance and explainability with a traditional machine learning (ML) model and large language model (LLM). A domain-specific noisy textual dataset of injury narratives had to be classified into six cause-of-injury codes. While the ML model was trained on pre-labelled injury narratives, LLM and humans did not receive any specialized training. The explainability of different approaches was compared using the words they focused on during classification. These words were identified using eye-tracking for humans, explainable AI approach LIME for ML model, and prompts for LLM. The classification performance of ML model was relatively better than LLM and humans- overall and particularly for complicated and challenging to classify narratives. The top-3 words used by ML and LLM for classification agreed with humans to a greater extent as compared to later words.

show abstract

Section: Related Work On Eye Trackingmentioning

confidence: 99%

Comparing Human Text Classification Performance and Explainability with Large Language and Machine Learning Models Using Eye-Tracking

Nanda,

Venkatesh,

Jaiswal

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…In addition to reading and language comprehension, eye-tracking has also been used in understanding how humans perform other linguistic tasks such as annotation 18,19 . Previous studies 20,21 have analyzed eye tracking data such as number of fixations, search time, and fixation duration recorded during manual annotation of named-entities in texts to a) estimate the difficulty level based on cognitive load assessed using gaze data eye-tracking data, and, b) identify prominent features used by humans for named-entity recognition.…”

Section: Related Work On Eye Trackingmentioning

confidence: 99%

Comparing human text classification performance and explainability with large language and machine learning models using eye-tracking

Divya Venkatesh,

Jaiswal,

Nanda

2024

Sci Rep

View full text Add to dashboard Cite

To understand the alignment between reasonings of humans and artificial intelligence (AI) models, this empirical study compared the human text classification performance and explainability with a traditional machine learning (ML) model and large language model (LLM). A domain-specific noisy textual dataset of 204 injury narratives had to be classified into 6 cause-of-injury codes. The narratives varied in terms of complexity and ease of categorization based on the distinctive nature of cause-of-injury code. The user study involved 51 participants whose eye-tracking data was recorded while they performed the text classification task. While the ML model was trained on 120,000 pre-labelled injury narratives, LLM and humans did not receive any specialized training. The explainability of different approaches was compared based on the top words they used for making classification decision. These words were identified using eye-tracking for humans, explainable AI approach LIME for ML model, and prompts for LLM. The classification performance of ML model was observed to be relatively better than zero-shot LLM and non-expert humans, overall, and particularly for narratives with high complexity and difficult categorization. The top-3 predictive words used by ML and LLM for classification agreed with humans to a greater extent as compared to later predictive words.

show abstract

Cognitive Signals of Language Processing

Beinborn,

Hollenstein

2023

Synthesis Lectures on Human Language Technologies

View full text Add to dashboard Cite

The ZuCo Benchmark on Cross-Subject Reading Task Classification with EEG and Eye-Tracking Data

Cited by 4 publications

References 78 publications

Comparing Human Text Classification Performance and Explainability with Large Language and Machine Learning Models Using Eye-Tracking

Comparing Human Text Classification Performance and Explainability with Large Language and Machine Learning Models Using Eye-Tracking

Comparing human text classification performance and explainability with large language and machine learning models using eye-tracking

Cognitive Signals of Language Processing

Contact Info

Product

Resources

About