Abstract:Contextualized, meaning-based interaction in the foreign language is widely recognized as crucial for second language acquisition. Correspondingly, current exercises in foreign language teaching generally require students to manipulate both form and meaning. For Intelligent Language Tutoring Systems to support such activities, they thus must be able to evaluate the appropriateness of the meaning of a learner response for a given exercise. We discuss such a content-assessment approach, focusing on reading comprehension exercises. We pursue the idea that a range of simultaneously available representations at different levels of complexity and linguistic abstraction provide a good empirical basis for content assessment. We show how an annotation-based NLP architecture implementing this idea can be realized and that it successfully performs on a corpus of authentic learner answers to reading comprehension questions. To support comparison and sustainable development on content assessment, we also define a general exchange format for such exercise data.Keywords: content assessment, shallow semantic analysis, meaning comparison, textual entailment, intelligent computer-assisted language learning, ICALL, intelligent tutoring systems Biographical notes: Detmar Meurers is a professor of Computational Linguistics at the University of Tübingen, Germany. Previously he was an associate professor at The Ohio State University, where he founded the ICALL research group focusing on intelligent tutoring systems, content assessment, and automatic input enhancement for language learners.Ramon Ziai is a PhD candidate at the Collaborative Research Center 833 at the University of Tübingen, Germany. His main research interest and background is in computational linguistics. More specifically, he is interested in shallow semantic analysis and the question of how ill-formed input can be processed.

We discuss the collection and analysis of a cross-sectional and longitudinal learner corpus consisting of answers to reading comprehension questions written by adult second language learners of German. We motivate the need for such task-based learner corpora and identify the properties which make reading comprehension exercises a particularly interesting task. In terms of the creation of the corpus, we introduce the web-based WELCOME tool we developed to support the decentralized data collection and annotation of the richly structured corpus in real-life language teaching programs. On the analysis side, we investigate the binary and the complex content-assessment classification scheme used by the annotators and the inter-annotator agreement obtained for the current corpus snapshot, at the halfway point of our four-year effort. We present results showing that for such task-based corpora, meaning assessment can be performed with reasonable agreement and we discuss several sources of disagreement.