No abstract

Despite consistent calls for authentic stimuli in listening tests for better construct representation, unscripted texts have been rarely adopted in high-stakes listening tests due to perceived inefficiency. This study details how a local academic listening test was developed using authentic unscripted audio-visual texts from the local target language use (TLU) domain without compromising the reliability of the test results and validity of the score interpretations. The purpose of the listening test was to identify international students who need additional language support at a U.S. university. We show that efficiency persists when using authentic unscripted texts that are representative of the local context both at the test development phase and at the classification phase where placement decisions are made in a dependable manner. Expert judgments highlighted the improved correspondence of the listening test using locally sourced audio-visual texts to the local TLU domain, providing additional support for using the listening test for local placement purposes. Additionally, dimensionality assessments demonstrated that test design decisions inevitably entailed with using authentic unscripted texts did not threaten the internal structure of the test. We argue that local resources are indispensable in developing authentic test stimuli and in supporting the validity of local test interpretation and use.

This paper presents the Duolingo English Test’s speaking construct, situated within the Duolingo English Test assessment ecosystem (Burstein et al., 2022). We describe how the Duolingo English Test defines, operationalizes, and measures speaking through various speaking-related item types. The operationalization and measurement of the speaking construct includes the item-type design process and automated item generation processes.

This paper introduces a new integrated task type on the Duolingo English Test called Interactive Listening and grounds the task within the Duolingo English Test’s theoretical language assessment design framework and its assessment ecosystem. The task and automated item generation methods contribute to measurement of the constructs of L2 listening, reading, and writing, thereby strengthening the validity claims of the Duolingo English Test.

Assessments, especially those used for high-stakes decision making, draw on evidence-based frameworks. Such frameworks inform every aspect of the testing process, from development to results reporting. The frameworks that language assessment professionals use draw on theory in language learning, assessment design, and measurement and psychometrics in order to provide underpinnings for the evaluation of language skills including speaking, writing, reading, and listening. This paper focuses on the construct, or underlying trait, of writing ability. The paper conceptualizes the writing construct for the Duolingo English Test, a digital-first assessment. “Digital-first” includes technology such as artificial intelligence (AI) and machine learning, with human expert involvement, throughout all item development, test scoring, and security processes. This work is situated in the Burstein et al. (2022) theoretical ecosystem for digital-first assessment, the first representation of its kind that incorporates design, validation/measurement, and security all situated directly in assessment practices that are digital first. The paper first provides background information about the Duolingo English Test and then defines the writing construct, including the purposes for writing. It also introduces principles underpinning the design of writing items and illustrates sample items that assess the writing construct.