An Application of Exploratory Data Analysis in the Development of Game-Based Assessments

DiCerbo, Kristen E.; Bertling, Maria; Stephenson, Shonté; Jia, Yunyi; Mislevy, Robert J.; Bauer, Malcolm; Jackson, G. Tanner

doi:10.1007/978-3-319-05834-4_14

Cited by 28 publications

(9 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By observing a sequence of actions, whether the player installs alternative forms of energy prior to bulldozing the coal plants or not, provides us evidence about whether the learner understands that the coal plants have multiple effects on the city, as opposed to just the single effect of air pollution. Gathering multiple observations of similar choices, allows us to build models of their levels of systems thinking (DiCerbo, Castellano, Jia, Mislevy, & Jin, 2015).…”

Section: Process Datamentioning

confidence: 99%

“…This insight can then help point to particular areas where a learner's solution might differ from the “prescribed” solution. Over time, and thousands or tens of thousands of users, we will be able to identify even less common solution paths in digital experiences, often with the help of exploratory data analysis (Behrens, DiCerbo, Yel, & Levy, 2012; DiCerbo et al., 2015). In the SimCityEDU game, over time it became apparent that a small percentage of players “solved” the air pollution problem by bulldozing the entire city.…”

Section: Process Datamentioning

confidence: 99%

See 1 more Smart Citation

Assessment for Learning with Diverse Learners in a Digital World

DiCerbo

2020

Educational Measurement

View full text Add to dashboard Cite

We have the ability to capture data from students' interactions with digital environments as they engage in learning activity. This provides the potential for a reimagining of assessment to one in which assessment become part of our natural education activity and can be used to support learning. These new data allow us to more closely examine the processes learners use to solve problems, not just their final solutions, so educators can be more targeted in their intervention. New capabilities may address some of the consistent criticisms of assessment, but they also present new dangers, particularly for diverse learners. Systems are in danger of replicating the biases from our non‐digital world in our digital tasks and in the creation of the scoring tools that accompany them.

show abstract

Section: Process Datamentioning

confidence: 99%

Section: Process Datamentioning

confidence: 99%

Assessment for Learning with Diverse Learners in a Digital World

DiCerbo

2020

Educational Measurement

View full text Add to dashboard Cite

show abstract

“…Thus an appropriate set of attributes and parameters for each state needs to be established for each activity in collaboration between the members of the interdisciplinary design team for the VPAs. In early stages of development, finer grained, lower level events may need to be included in the ETF so patterns among them can be further explored to determine exactly which actions and system state variables are needed to produce the required evidence (e.g., DiCerbo et al, ; Halverson & Owen, ; Roberts, Chung, & Parks, ).The subsequent operational version of the ETF may then need to contain only higher level computed events, if the required computation is straightforward to calculate.…”

Section: Evidence Trace Filementioning

confidence: 99%

“…SimCity verb clauses contain information that provides evidence about the level of systems thinking represented by a student's play in a challenge, but verb clauses alone need further analysis to determine just what that evidence is. DiCerbo et al () describe some of the analyses carried out with data at the verb‐clause and heartbeat level to determine patterns of verb‐clause events that provide evidence of students' understanding and strategies. These analyses were exploratory investigations at the process level labeled “Pattern recognition processes.” Their finalized forms were summarized in the form of values of observable variables, indicated as (6) in Figure .…”

Section: Use Case For An Evidence Trace File In Simcityedu: Pollutionmentioning

confidence: 99%

The Evidence Trace File: A Data Structure for Virtual Performance Assessments Informed by Data Analytics and Evidence‐Centered Design

Hao

Mislevy

2018

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

Virtual performance assessments (VPAs), such as game‐ and simulation‐based assessments, provide promising ways for assessing complex and integrated skills. However, the high cost, long development cycle, and complex scoring process significantly hinder the adoption of VPAs, particularly in large‐scale assessments with tight deadlines and limited budgets. One significant impediment is the lack of consensus about how to capture and store evidence‐bearing process data from each VPA session into a format that facilitates quality checks, scoring, and analysis. In current practice, data from VPAs are generally stored in logs, either as standalone files or as a set of entries in a database. Programmers have long used logs to save process information about software systems for monitoring and debugging the software system itself. These purposes are fundamentally different from the purpose of assessment, which aims to capture evidence to support the measurement of a test taker's proficiency with respect to targeted constructs. Collapsing information that might be needed for the two purposes into a single log file can lead to confusion, even confrontation, among VPA designers, software developers, and scoring experts. To mitigate the problem, in this paper we introduce the notion of a structured evidence trace file (ETF) as distinguished from the familiar but idiosyncratic term, log file. The ETF is intended to express captured telemetry data in a way that meets the needs from the assessment perspective. Its specification as a standalone deliverable allows VPA designers and scoring experts to cleanly define and assure the quality of the information they need from VPA sessions in forms they find most useful for assessment purposes without adding additional complexity or structures for the information that software developers need for debugging purposes. In turn, software developers will be clearly informed about what information is expected from the assessment perspective and can manage the data as they choose as long as they can deliver the specified ETF. In addition to its application in VPAs, the ETF structure can also be employed more generally with digitally based assessments (DBAs) in which response‐process data are captured and used for assessment purposes, such as timing information, response changes, and tool usage in multiple‐choice tests. As a known data structure, it can sometimes also be employed for additional purposes in VPA analysis, such as transmitting information across phases of multiple‐stage analysis.

show abstract

“…They become meaningful as moves through the syntax of the game environment, then meaningful semantically as higher level actions such as bulldozing a high-pollution power plant. Sequences and patterns of these actions serve as evidence of strategies that reflect a player's thinking about the underlying system (e.g., constructing a new low-pollution power plant before bulldozing the old one in order to maintain power, and therefore businesses and jobs; DiCerbo et al, 2015). Counts and instances of higher level actions are the OVs, input to a Bayes net psychometric model with an SMV variable based on a systems-thinking learning progression (Bertling & Castellano, 2016).…”

Section: Challenge 6: How Do We "Score" Complex Performances At Scale?mentioning

confidence: 99%

How Developments in Psychology and Technology Challenge Validity Argumentation

Mislevy

2016

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

Validity is the sine qua non of properties of educational assessment. While a theory of validity and a practical framework for validation has emerged over the past decades, most of the discussion has addressed familiar forms of assessment and psychological framings. Advances in digital technologies and in cognitive and social psychology have expanded the range of purposes, targets of inference, contexts of use, forms of activity, and sources of evidence we now see in educational assessment. This article discusses some of these developments and how concepts and representations that are employed to design and use assessments, hence to frame validity arguments, can be extended accordingly. Ideas are illustrated with a variety of examples, with an emphasis on assessment in higher education.

show abstract

An Application of Exploratory Data Analysis in the Development of Game-Based Assessments

Cited by 28 publications

References 10 publications

Assessment for Learning with Diverse Learners in a Digital World

Assessment for Learning with Diverse Learners in a Digital World

The Evidence Trace File: A Data Structure for Virtual Performance Assessments Informed by Data Analytics and Evidence‐Centered Design

How Developments in Psychology and Technology Challenge Validity Argumentation

Contact Info

Product

Resources

About