Examining Linguistic Characteristics of Paraphrase in Test‐taker Summaries

Abstract

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. AbstractThis annotation study is designed to help us gain an increased understanding of paraphrase strategies used by native and nonnative English speakers and how these strategies might affect test takers' essay scores. Toward that end, this study aims to examine and analyze the paraphrase and the types of linguistic modifications used in paraphrase in test-taker responses and differences that may exist between native and nonnative English speakers. We are also interested in how these factors might influence final essay score. Outcomes discussed in this report can be used to inform the development of new e-rater ® scoring engine features that capture information related to paraphrase, specifically in nonnative speaker responses to the TOEFL ® exam integrated writing task.

Content Importance Models for Scoring Writing From Sources

“…The estimation of use-in-a-corpus is more accurate for smaller n, because longer ngrams are more susceptible to paraphrasing, which leads to under-estimation of use. Assuming that paraphrasing behavior of good and bad writers is not the same -in fact, there is corpus evidence that better writers paraphrase more (Burstein et al, 2012) -the resulting inaccuracies might impact the estimation of differential use in a systematic manner, making the n > 1 models less effective than the unigrams. Given that (a) the GoodVsBad bigram model is the second best overall in spite of the shortcomings of the estimation process, and (b) that the bigram models worked better than unigram models for all the other content importance models, the GoodVsBad bigram model could probably be improved significantly by using a more flexible information realization mechanism.…”

Integrated Second Language Writing Assessment: Why? What? How?

“…In a study about paraphrasing in TOEFL iBT integrated writing, Burnstein et al () delved more deeply into the nature of the integration of source material using computational tools to explore paraphrasing. They discovered a difference in paraphrases from the listening source materials and from reading.…”