Document similarity measures to distinguish native vs. non-native essay writers

Abstract

The ability to distinguish statistically different populations of speakers or writers can be an important asset in many NLP applications. In this paper, we describe a method of using document similarity measures to describe differences in behavior between native and non-native speakers of English in a writing task. 1

Examining Linguistic Characteristics of Paraphrase in Test‐taker Summaries

“…Tables 6 and 7 illustrate that the number of paraphrases drawn from the lecture stimuli were positively correlated with essay score, and these correlations were even stronger with the nonnative speaker data. This finding is consistent with Gurevich and Deane (2007), who also found for the same data set that summaries with more evidence of paraphrase drawn from the lecture stimuli received higher scores. These consistent findings may be possibly due to the fact that the lecture is unavailable for the full duration of the exam, and so paraphrase from the lecture stimuli relies on accurate note-taking during the lecture portion and recall from memory.…”

“…Consistent with the discussion above, this finding also suggests that compared to the native speaker responses, nonnative speaker responses appear to have been more harshly penalized for paraphrasing from the reading but more strongly rewarded for paraphrasing from the lecture. These findings are consistent with outcomes in Gurevich and Deane (2007). Using the same data, Gurevich and Deane performed a comparison of language in test-taker responses and their similarity to language in the reading and lecture portions of the stimuli.…”

“…The existing TOEFL integrated data set that was used contained native and nonnative English speaker responses (see Gurevich & Deane, 2007). All data are summaries in response to one prompt (see Appendix A).…”

Content Importance Models for Scoring Writing From Sources

“…Our results are related to the findings of Gurevich and Deane (2007) who studied the difference between the reading and the lecture in their impact on essay scores for this test. Using data from a single prompt, they showed that the difference between the essay's average cosine similarity to the reading and its average cosine similarity to the lecture is predictive of the score for non-native speakers of English, thus using a model similar to LectVsRead, although they took all lecture, reading, and essay words into account, in contrast to our model that looks only at n-grams that appear in the lecture.…”

A Corpus of Non-Native Written English Annotated for Metaphor

“…Research in automated assessment of students' writing, both native and non-native, is increasingly moving beyond traditional models that emphasize English conventions, sophistication of vocabulary, and organization (Attali and Burstein, 2006). Assessing aspects of content is a rapidly growing research topic, including evaluation of arguments, of the writer's use of information from source materials, of the coherence of the essay, among others (Ghosh et al, 2016;Persing and Ng, 2015;Stab and Gurevych, 2014;Song et al, 2014;Somasundaran et al, 2014;Gurevich and Deane, 2007). Use of metaphor is another aspect of language use that goes beyond grammar and mechanics; recent research suggests that use of metaphor differs with proficiency (Beigman Klebanov et al, 2013), including in non-native writing (Littlemore et al, 2013).…”