Automated Measures of Specific Vocabulary Knowledge from Constructed Responses ('Use These Words to Write a Sentence Based on this Picture')

Abstract

We describe a system for automatically scoring a vocabulary item type that asks test-takers to use two specific words in writing a sentence based on a picture. The system consists of a rule-based component and a machine learned statistical model which uses a variety of construct-relevant features. Specifically, in constructing the statistical model, we investigate if grammar, usage, and mechanics features developed for scoring essays can be applied to short answers, as in our task. We also explore new features reflecting the quality of the collocations in the response, as well as features measuring the consistency of the response to the picture. System accuracy in scoring is 15 percentage points greater than the majority class baseline and 10 percentage points less than human performance.

Shallow Semantic Reasoning from an Incomplete Gold Standard for Learner Language

“…In research closer to our own image-based work, Somasundaran and Chodorow (2014) analyze learner responses to a PDT where the responses were constrained by requiring the use of specific words. The pictures were annotated by experts, and the relevance of responses was calculated through the overlap of the response and annotation contents.…”

“…Although much current work on analyzing learner language focuses on grammatical error detection and correction (e.g., Leacock et al, 2014), there is a growing body of work covering varying kinds of semantic analysis (e.g., Meurers et al, 2011;Bailey and Meurers, 2008;Dickinson, 2014, 2013;Petersen, 2010), including assessment-driven work (e.g., Somasundaran et al, 2015;Somasundaran and Chodorow, 2014). One goal of such work is to facilitate intelligent language tutors (ILTs) and language assessment tools that maximize communicative interaction, as suggested by research in second language instruction (cf.…”

Automated Scoring of Picture-based Story Narration

“…In order to test if a given response tells a story that is relevant to the pictures in the prompt, we calculate the overlap of the content of the response and the content of the pictures similar to (Somasundaran and Chodorow, 2014). To facilitate this, each prompt is associated with a reference corpus containing a detailed description of each picture, and also an overall narrative that ties together the events in the pictures.…”

“…Our work explores the other (complementary) dimensions of the test such as language use, content relevance and story development. Somasundaran and Chodorow (2014) construct features for awkward word usage and content relevance for a written vocabulary test which we adapt for our task. Discourse organization features have been employed for essay scoring of written essays in the expository and argumentative genre (Attali and Burstein, 2006).…”

Annotating picture description task responses for content analysis

“…In our own previous work Dickinson, 2016, 2013), we annotated a small set of PDT responses as correct or incorrect, with incorrect responses further labeled as errors of form or meaning. Somasundaran and Chodorow (2014) presented work on PDT re-sponses in which respondents used provided vocabulary words. Responses were manually annotated on a holistic four point scale, and a set of five features (relating to meaning, relevance and language use) were calculated based on statistical assumptions.…”