EPCAL: ETS Platform for Collaborative Assessment and Learning

Hao, Jiangang; Liu, Lei; Davier, Alina A. von; Lederer, Nathan; Zapata‐Rivera, Diego; Jakl, Peter; Bakkenson, Michael

doi:10.1002/ets2.12181

Cited by 11 publications

(9 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Though some statistically significant effects have been detected at the subgroup level (thanks to a large number of participants), it does not allow us to reveal more details of each team member's process. In ongoing work, we have collected new data using a task hosted on the ETS Platform for Collaborative Assessment and Learning (Hao et al, 2017c). The new task elicits over 120 turns of communication in each team.…”

Section: Discussionmentioning

confidence: 99%

Characterizing Interactive Communications in Computer-Supported Collaborative Problem-Solving Tasks: A Conditional Transition Profile Approach

Hao

Mislevy

2019

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Communication in a collaborative problem-solving activity plays a pivotal role in the success of the collaboration in both academia and the workplace. Computer-supported collaboration makes it possible to collect large-scale communication data to investigate the process at a finer granularity. In this paper, we introduce a conditional transition profile (CTP) to characterize aspects of each team member's communication. Based on the data from a large-scale empirical study, we found that participants in the same team tend to show similar CTP compared to participants from different teams. We also found that team members who showed more “negotiation” after the partner “shared” information tended to show more improvement after the collaboration while those who continued sharing ideas while their partners were negotiating tended to improve less.

show abstract

Section: Discussionmentioning

confidence: 99%

Characterizing Interactive Communications in Computer-Supported Collaborative Problem-Solving Tasks: A Conditional Transition Profile Approach

Hao

Mislevy

2019

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…This kind of system needs to know not only what a desired state looks like but also how to get there. An example of a guiding system that makes a first step in this direction is the EPCAL platform (Hao et al, 2017), where an intelligent facilitator provides high-level prompts for group members to stay on topic and be polite, engaged, and responsive to others when they are perceived to be lacking in these areas.…”

Section: Opportunities and Challenges For The Field Of Collaboration Analyticsmentioning

confidence: 99%

“…The final two rows describe the main area of contribution and examples of systems that have implemented some of these features. EPCAL (Hao at al., 2017)…”

Section: What Are the Potential Futures Of Collaboration Analytics?mentioning

confidence: 99%

Collaboration Analytics — Current State and Potential Futures

Schneider

Dowell

Thompson

2021

JLA

View full text Add to dashboard Cite

This special issue brings together a rich collection of papers in collaboration analytics. With topics including theory building, data collection, modelling, designing frameworks, and building machine learning models, this issue represents some of the most active areas of research in the field. In this editorial, we summarize the papers; discuss the nature of collaboration analytics based on this body of work; describe the associated opportunities, challenges, and risks; and depict potential futures for the field. We conclude by discussing the implications of this special issue for collaboration analytics.

show abstract

“…Moreover, as assessment designers can specify the format of the ETF, it is possible to develop cross‐task and cross‐assessment data analytics tools that make the data analyses more efficient and cost‐effective. For example, such a workflow was implemented in the development of the EPCAL (Hao et al, ) and in the glassPy analytics solution (Hao et al, ).…”

Section: Use Cases For Evidence Trace Files In Hydrivementioning

confidence: 99%

“…In the next section, we locate the ETF with respect to the ECD proficiency, evidence, and task models. In the remaining sections, we take a closer look at how an ETF can be implemented, using examples from selected VPAs and a digital platform for delivering collaborative tasks (ETS Platform for Collaborative Assessment and Learning [EPCAL], Hao et al, ).The final section provides closing comments.…”

mentioning

confidence: 99%

The Evidence Trace File: A Data Structure for Virtual Performance Assessments Informed by Data Analytics and Evidence‐Centered Design

Hao

Mislevy

2018

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

Virtual performance assessments (VPAs), such as game‐ and simulation‐based assessments, provide promising ways for assessing complex and integrated skills. However, the high cost, long development cycle, and complex scoring process significantly hinder the adoption of VPAs, particularly in large‐scale assessments with tight deadlines and limited budgets. One significant impediment is the lack of consensus about how to capture and store evidence‐bearing process data from each VPA session into a format that facilitates quality checks, scoring, and analysis. In current practice, data from VPAs are generally stored in logs, either as standalone files or as a set of entries in a database. Programmers have long used logs to save process information about software systems for monitoring and debugging the software system itself. These purposes are fundamentally different from the purpose of assessment, which aims to capture evidence to support the measurement of a test taker's proficiency with respect to targeted constructs. Collapsing information that might be needed for the two purposes into a single log file can lead to confusion, even confrontation, among VPA designers, software developers, and scoring experts. To mitigate the problem, in this paper we introduce the notion of a structured evidence trace file (ETF) as distinguished from the familiar but idiosyncratic term, log file. The ETF is intended to express captured telemetry data in a way that meets the needs from the assessment perspective. Its specification as a standalone deliverable allows VPA designers and scoring experts to cleanly define and assure the quality of the information they need from VPA sessions in forms they find most useful for assessment purposes without adding additional complexity or structures for the information that software developers need for debugging purposes. In turn, software developers will be clearly informed about what information is expected from the assessment perspective and can manage the data as they choose as long as they can deliver the specified ETF. In addition to its application in VPAs, the ETF structure can also be employed more generally with digitally based assessments (DBAs) in which response‐process data are captured and used for assessment purposes, such as timing information, response changes, and tool usage in multiple‐choice tests. As a known data structure, it can sometimes also be employed for additional purposes in VPA analysis, such as transmitting information across phases of multiple‐stage analysis.

show abstract

EPCAL: ETS Platform for Collaborative Assessment and Learning

Cited by 11 publications

References 11 publications

Characterizing Interactive Communications in Computer-Supported Collaborative Problem-Solving Tasks: A Conditional Transition Profile Approach

Characterizing Interactive Communications in Computer-Supported Collaborative Problem-Solving Tasks: A Conditional Transition Profile Approach

Collaboration Analytics — Current State and Potential Futures

The Evidence Trace File: A Data Structure for Virtual Performance Assessments Informed by Data Analytics and Evidence‐Centered Design

Contact Info

Product

Resources

About