How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items

Abstract

Log data from educational assessments attract more and more attention and largescale assessment programs have started providing log data as scientific use files. Such data generated as a by-product of computer-assisted data collection has been known as paradata in survey research. In this paper, we integrate log data from educational assessments into a taxonomy of paradata. To provide a generic framework for the analysis of log data, finite state machines are suggested. Beyond its computational value, the specific benefit of using finite state machines is achieved by separating platform-specific log events from the definition of indicators by states. Specifically, states represent filtered log data given a theoretical process model, and therefore, encode the information of log files selectively. The approach is empirically illustrated using log data of the context questionnaires of the Programme for International Student Assessment (PISA). We extracted item-level response time components from questionnaire items that were administered as item batteries with multiple questions on one screen and related them to the item responses. Finally, the taxonomy and the finite state machine approach are discussed with respect to the definition of complete log data, the verification of log data and the reproducibility of log data analyses.

Validating process variables of sourcing in an assessment of multiple document comprehension

“…Log‐files record process data collected during the processing of a computer‐based task, such as mouse clicks with timestamps. These log events can be used to reconstruct the test‐taking process (Kroehne & Goldhammer, ), allowing different representations of sourcing behaviours to be captured (Table ).…”

“…Sourcing is a critical aspect required for following a discourse, for example, when readers attempt to create a comprehensive understanding of a topic from multiple perspectives. This study therefore uses a standardized assessment of multiple document comprehension (MDC) to evaluate whether meaningful episodes (‘states’; Kroehne & Goldhammer, ) of sourcing occur as expected by current theories on the functions of sourcing. These episodes are examined for their relations to characteristics of persons, documents, tasks, and the administration of the MDC assessment, to validate their intended meaning and reduce ambiguities in their interpretation (Goldhammer & Zehner, ).…”

Invariance of the Response Processes Between Gender and Modes in an Assessment of Reading

“…The strategy described in this paper requires partitioning the whole testing time into segments, which start and end with the selection of answers in consecutive questions. The necessary theoretical justification for the treatment of the self-selected order of responses can be provided by a general framework that uses log data to distinguish meaningful states of the testtaking process [27]. In this framework, log events are processed algorithmically by reconstructing the sequence of states using, for instance, a finite state machine.…”

“…The use of the finite state machine approach for analyzing log data assumes that the test-taking process can be described as a progression of states that corresponds to one question at a time, starting with the reading text and the first question for each unit. By starting the finite state machine used to reconstruct the sequence of states for each test-taker in the state "Reading Unit Text & Answering Q1, " knowledge about the CBA and PBA instrument is included in the analysis using the finite state machine approach introduced in [27].…”

Introduction to the Vol.45, No.2, 2018

“…In this issue, two special features were included:"Advanced Methodologies for Bayesian Networks" (Scutari 2018;Capdevila et al 2018;Peña 2018, Sugaya et al 2018 which was edited by Joe Suzuki, Antti Hytthinen, and Brandon Malone, and "Advanced Technologies in Educational Assessment" (Deonovic et al 2018, Slater and Baker 2018, Frey 2018, Kroehne and Goldhammer 2018, Specifically, de Klerk et al 2018, Nguyen et al2018, Shi et al 2018, Rights et al 2018 which was edited by Ronny Scherer and Marie Wiberg. Both issues specifically addressed state-of-theart methods of statistics, machine learning, artificial intelligence and data science.…”