Is this Sentence Difficult? Do you Agree?

Brunato, Dominique; Mattei, Lorenzo De; Dell’Orletta⋄, Felice; Iavarone, Benedetta; Venturi, Giulia

doi:10.18653/v1/d18-1289

Cited by 32 publications

(35 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We first collected an appropriate corpus to evaluate the effect of context on the perception of sentence complexity. We started from the crowdsourced dataset by Brunato et al (2018), which contains 1, 200 sentences annotated for perceived complexity on a 7-point Likert scale. We similarly built a crowdsourcing task, asking native English speakers to read each sentence and rate its complexity on the same scale.…”

Section: Approachmentioning

confidence: 99%

“…From a human-based perspective, sentence complexity is assessed by measures of processing effort or performance in behavioral tasks. In this respect, a large part of studies has focused on reading single sentences and correlating syntactic and lexical properties with observed difficulty, being it captured by cognitive signals, such as eye-tracking metrics (Rayner, 1998;King and Just, 1991), or by explicit judgments of complexity given by readers (Brunato et al, 2018). However, models of language comprehension underline the importance of contextual cues, such as the presence of explicit cohesive devices, in building a coherent representation of a text (Kintsch et al, 1975;McNamara, 2001).…”

Section: Introductionmentioning

confidence: 99%

“…Our study aims to understand how the context surrounding a sentence influences its 'perceived' complexity by humans. As we consider linguistic complexity from the individual's perspective, following Brunato et al 2018, we use the term complexity as a synonym of difficulty. Also, we assume that sentence complexity is a gradient rather than a binary concept and we operationalize perceived complexity as a score on an ordinal scale.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Sentence Complexity in Context

Iavarone¹,

Brunato²,

Dell’Orletta⋄³

2021

Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics

Self Cite

View full text Add to dashboard Cite

We study the influence of context on how humans evaluate the complexity of a sentence in English. We collect a new dataset of sentences, where each sentence is rated for perceived complexity within different contextual windows. We carry out an in-depth analysis to detect which linguistic features correlate more with complexity judgments and with the degree of agreement among annotators. We train several regression models, using either explicit linguistic features or contextualized word embeddings, to predict the mean complexity values assigned to sentences in the different contextual windows, as well as their standard deviation. Results show that models leveraging explicit features capturing morphosyntactic and syntactic phenomena perform always better, especially when they have access to features extracted from all contextual sentences.

show abstract

Section: Approachmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Sentence Complexity in Context

Iavarone¹,

Brunato²,

Dell’Orletta⋄³

2021

Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recently, (Stajner et al, 2017) and evaluated the RPSL task with a huge dataset -the Newsela dataset, comprising 550 thousand sentences, three times greater than Wikipedia-SimpleWikipedia. (Brunato et al, 2018) evaluated the perception of complexity and agreement between annotators, while (Timm, 2018) investigated automatic sentence simplifications, using eye-tracking tools.…”

Section: Readability Prediction At Sentence Levelmentioning

confidence: 99%

Using Eye-tracking Data to Predict the Readability of Brazilian Portuguese Sentences in Single-task, Multi-task and Sequential Transfer Learning Approaches

Leal

Vieira²,

Rodrigues³

et al. 2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Sentence complexity assessment is a relatively new task in Natural Language Processing. One of its aims is to highlight in a text which sentences are more complex to support the simplification of contents for a target audience (e.g., children, cognitively impaired users, non-native speakers and low-literacy readers ). This task is evaluated using datasets of pairs of aligned sentences including the complex and simple version of the same sentence. For Brazilian Portuguese, the task was addressed by (Leal et al., 2018), who set up the first dataset to evaluate the task in this language, reaching 87.8% of accuracy with linguistic features. The present work advances these results, using models inspired by (Gonzalez-Garduño and Søgaard, 2018), which hold the state-of-the-art for the English language, with multi-task learning and eyetracking measures. First-Pass Duration, Total Regression Duration and Total Fixation Duration were used in two moments; first to select a subset of linguistic features and then as an auxiliary task in the multi-task and sequential learning models. The best model proposed here reaches the new state-of-the-art for Portuguese with 97.5% accuracy 1 , an increase of almost 10 points compared to the best previous results, in addition to proposing improvements in the public dataset after analysing the errors of our best model. 1 Accuracy in our task is how close the model is to the true value, when assessing whether a given sentence is simple or complex, in a 10-fold cross-validation test.

show abstract

“…One of the most well-known measure along these lines is the Flesch-Kincaid readability index (Kincaid et al, 1975), which combines these two measures into a global score. This approach has recently been renewed by the use of supervised statistical learning methods capable of integrating into the prediction of readability a very large number of linguistic characteristics (Schwarm and Ostendorf, 2005;Petersen and Ostendorf, 2009;Vajjala and Meurers, 2012;François and Fairon, 2012;Vajjala and Meurers, 2014;Brunato et al, 2018) aimed at capturing readability indices at the lexical, syntactic, semantic and even discursive levels. It can be argued that these enhanced feature sets are able to take into account so-called cognitive factors (Feng et al, 2009).…”

Section: Introductionmentioning

confidence: 99%

Measuring text readability with machine comprehension: a pilot study

Benzahra¹,

Yvon²

2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

This article studies the relationship between text readability levels and automatic machine understanding systems. Our hypothesis is that the simpler a text is, the better it should be understood by a machine. We thus expect a strong correlation between readability levels on the one hand, and performance of automatic reading systems on the other hand. We test this hypothesis with several understanding systems based on language models of varying strengths, measuring this correlation on two corpora of journalistic texts. Our results suggest that this correlation is quite small and that existing comprehension systems are far to reproduce the gradual improvement of their performance on texts of decreasing complexity.

show abstract

Is this Sentence Difficult? Do you Agree?

Cited by 32 publications

References 14 publications

Sentence Complexity in Context

Sentence Complexity in Context

Using Eye-tracking Data to Predict the Readability of Brazilian Portuguese Sentences in Single-task, Multi-task and Sequential Transfer Learning Approaches

Measuring text readability with machine comprehension: a pilot study

Contact Info

Product

Resources

About