Taming Log Files From Game/Simulation‐Based Assessments: Data Models and Data Analysis Tools

Self Cite

Virtual performance assessments (VPAs), such as game‐ and simulation‐based assessments, provide promising ways for assessing complex and integrated skills. However, the high cost, long development cycle, and complex scoring process significantly hinder the adoption of VPAs, particularly in large‐scale assessments with tight deadlines and limited budgets. One significant impediment is the lack of consensus about how to capture and store evidence‐bearing process data from each VPA session into a format that facilitates quality checks, scoring, and analysis. In current practice, data from VPAs are generally stored in logs, either as standalone files or as a set of entries in a database. Programmers have long used logs to save process information about software systems for monitoring and debugging the software system itself. These purposes are fundamentally different from the purpose of assessment, which aims to capture evidence to support the measurement of a test taker's proficiency with respect to targeted constructs. Collapsing information that might be needed for the two purposes into a single log file can lead to confusion, even confrontation, among VPA designers, software developers, and scoring experts. To mitigate the problem, in this paper we introduce the notion of a structured evidence trace file (ETF) as distinguished from the familiar but idiosyncratic term, log file. The ETF is intended to express captured telemetry data in a way that meets the needs from the assessment perspective. Its specification as a standalone deliverable allows VPA designers and scoring experts to cleanly define and assure the quality of the information they need from VPA sessions in forms they find most useful for assessment purposes without adding additional complexity or structures for the information that software developers need for debugging purposes. In turn, software developers will be clearly informed about what information is expected from the assessment perspective and can manage the data as they choose as long as they can deliver the specified ETF. In addition to its application in VPAs, the ETF structure can also be employed more generally with digitally based assessments (DBAs) in which response‐process data are captured and used for assessment purposes, such as timing information, response changes, and tool usage in multiple‐choice tests. As a known data structure, it can sometimes also be employed for additional purposes in VPA analysis, such as transmitting information across phases of multiple‐stage analysis.

Section: Evidence Trace Filementioning

confidence: 99%

Section: Evidence Trace Filementioning

confidence: 99%

Section: Evidence Trace Filementioning

confidence: 99%

Section: Use Cases For Evidence Trace Files In Hydrivementioning

confidence: 99%

See 2 more Smart Citations

The Evidence Trace File: A Data Structure for Virtual Performance Assessments Informed by Data Analytics and Evidence‐Centered Design

Hao

Mislevy

2018

Self Cite

“…In the rest of this report, we introduce the details of the design and development of EPCAL. It is worth noting that in addition to these features, EPCAL also comes with integrated analytics support as part of ETS's process-data analytics solution (Hao, Smith, Mislevy, von Davier, & Bauer, 2016).…”

Section: Important Features Of a Platform For Studying Collaborationmentioning

confidence: 99%

EPCAL: ETS Platform for Collaborative Assessment and Learning

Hao

Liu

Davier

et al. 2017

Self Cite

Most existing software tools for online collaboration are designed to support the collaboration itself instead of the study of collaboration with a systematic team and task management system. In this report, we identify six important features for a platform to facilitate the study of online collaboration. We then introduce the Educational Testing Service Platform for Collaborative Assessment and Learning (EPCAL), which was developed to encompass these features to enable both large‐scale online collaboration itself and the research on these collaborations. The collaboration management features of the EPCAL also make it an ideal tool for teachers to carry out computer‐supported collaborative learning and assessment activities in classroom settings.

Analysis of Keystroke Sequences in Writing Logs

Zhu

Zhang

Deane

2019

The research on using event logs and item response time to study test‐taking processes is rapidly growing in the field of educational measurement. In this study, we analyzed the keystroke logs collected from 761 middle school students in the United States as they completed a persuasive writing task. Seven variables were extracted from the keystroke logs and compared with different score and gender groups. Group comparisons were also made using methodologies borrowed from sequence mining. Students' composition strategies over the course of the writing process were also investigated. The findings of this study have implications for gaining deeper understanding of observed group differences and for designing interventions to close the achievement gaps among population groups.