Using Networks to Visualize and Analyze Process Data for Educational Assessment

Abstract

New technology enables interactive and adaptive scenario-based tasks (SBTs) to be adopted in educational measurement. At the same time, it is a challenging problem to build appropriate psychometric models to analyze data collected from these tasks, due to the complexity of the data. This study focuses on process data collected from SBTs. We explore the potential of using concepts and methods from social network analysis to represent and analyze process data. Empirical data were collected from the assessment of Technology and Engineering Literacy, conducted as part of the National Assessment of Educational Progress. For the activity sequences in the process data, we created a transition network using weighted directed networks, with nodes representing actions and directed links connecting two actions only if the first action is followed by the second action in the sequence. This study shows how visualization of the transition networks represents process data and provides insights for item design. This study also explores how network measures are related to existing scoring rubrics and how detailed network measures can be used to make intergroup comparisons.

Examining the patterns of communication and connections among engineering professional skills in group discussion: A network analysis approach

“…Other studies applied SNA methods to understand the structures and features of non-human networks. For instance, researchers analyzed the networks of AOI (Areas of Interest) using data collected via tracking students' eyeball movements while answering science items in an onlinedelivered assessment [21]; and the transition networks of activities using action-log data recorded while students worked on complex online-delivered items [22]. In this study, we explore the usefulness of SNA in engineering education, specifically in terms of how SNA can assist in forming a deeper understanding of the communication processes and connections among EPSs evidenced through student-group discussions, for which information can be used as feedback to improve teaching and learning practices.…”

Network Analysis of Conversation Data for Engineering Professional Skills Assessment

“…In addition to connections among students, researchers have also used SNA to model the interactions among teachers, teaching coaches, and administrators to understand, for example, the impacts of interventions on school systems (Sweet, Thomas, & Junker, ). SNA has also been used to model nonhuman systems in education, such as the transitions among student activities recorded in the log data for simulation‐based assessment (Zhu, Shu, & von Davier, ) and students' eye movements while they were solving mathematics problems in scenario‐based assessment (Zhu & Feng, ). In both studies, network statistics were found to be closely related to student performance.…”

Data Mining Techniques in Analyzing Process Data: A Didactic

“…To take the temporal information into account, hierarchical vectorization of the rank ordered time intervals and the time interval distribution of event pairs were also introduced. In addition to these common analytic techniques, other existing data analytic methods for process data are Social Network Analysis (SNA; Zhu et al, 2016 ), Bayesian Networks/Bayes nets (BNs; Levy, 2014 ), Hidden Markov Model (Jeong et al, 2010 ), Markov Item Response Theory (Shu et al, 2017 ), diagraphs (DiCerbo et al, 2011 ) and process mining (Howard et al, 2010 ). Further, modern data mining techniques, including cluster analysis, decision trees, and artificial neural networks, have been used to reveal useful information about students' problem-solving strategies in various technology-enhanced assessments (e.g., Soller and Stevens, 2007 ; Kerr et al, 2011 ; Gobert et al, 2012 ).…”