Analyzing Collaborative Interactions Across Domains and Settings: An Adaptable Rating Scheme

Rummel, Nikol; Deiglmayr, Anne; Spada, Hans; Kahrimanis, George; Avouris, Nikolaos

doi:10.1007/978-1-4419-7710-6_17

Cited by 17 publications

(17 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ML models built in MMLA research to estimate discrete levels of collaboration quality also fail to offer additional information on the underlying reasons for the estimated quality level, thus offering fewer clues about how to scaffold students when collaborating (i.e., the actionability of the MMLA estimations is limited). This case study employs regression models to estimate collaboration quality on a continuous scale, and it also attempts to estimate various collaboration quality sub-dimensions [ 51 ] (argumentation, collaboration flow, knowledge exchange, sustaining mutual understanding, cooperative orientation, structuring problem-solving process and time management, individual task orientation) to offer more actionable estimation results.…”

Section: Illustrative Case Studymentioning

confidence: 99%

“…To obtain a ground-truth measure of collaboration quality, our research team manually labeled the videos from each group of collaborating students. We used the rating handbook created by Rummel et al [ 51 ] (which is an adaptable version of Meier et al’s rating scheme [ 59 ]) to quantify the groups’ collaboration quality for each 30 s window. This rating scheme involves seven dimensions of collaboration quality: sustaining mutual understanding (SMU), knowledge exchange (KE), argumentation (ARG), collaboration flow (CF), cooperative orientation (CO), structuring problem-solving process and time management (SPST) and individual task orientation (ITO).…”

Section: Illustrative Case Studymentioning

confidence: 99%

“…Two researchers used the rating handbook [ 51 ] as training material and independently coded 10 min of video from collaboration activities. After the first iteration, Cohen’s kappa was computed and found to be below an acceptable level (

< 0.60 for all dimensions).…”

Section: Illustrative Case Studymentioning

confidence: 99%

See 2 more Smart Citations

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

Chejara

Prieto

Ruiz‐Calleja

et al. 2021

Sensors

View full text Add to dashboard Cite

Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.

show abstract

Section: Illustrative Case Studymentioning

confidence: 99%

Section: Illustrative Case Studymentioning

confidence: 99%

See 1 more Smart Citation

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

Chejara

Prieto

Ruiz‐Calleja

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…We used Rummel et al's [11] collaboration quality rating scheme (itself adapted from [9]), assigning a collaboration quality score along seven dimensions 5 . We decided to use the adaptable version instead of the original scheme due to its applicability to a variety of CSCL settings.…”

Section: Data Annotationmentioning

confidence: 99%

Quantifying Collaboration Quality in Face-to-Face Classroom Settings Using MMLA

Chejara

Prieto

Ruiz‐Calleja

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The estimation of collaboration quality using manual observation and coding is a tedious and difficult task. Researchers have proposed the automation of this process by estimation into few categories (e.g., high vs. low collaboration). However, such categorical estimation lacks in depth and actionability, which can be critical for practitioners. We present a case study that evaluates the feasibility of quantifying collaboration quality and its multiple sub-dimensions (e.g., collaboration flow) in an authentic classroom setting. We collected multimodal data (audio and logs) from two groups collaborating face-to-face and in a collaborative writing task. The paper describes our exploration of different machine learning models and compares their performance with that of human coders, in the task of estimating collaboration quality along a continuum. Our results show that it is feasible to quantitatively estimate collaboration quality and its sub-dimensions, even from simple features of audio and log data, using machine learning. These findings open possibilities for in-depth automated quantification of collaboration quality, and the use of more advanced features and algorithms to get their performance closer to that of human coders.

show abstract

“…We performed a content analysis of all the contributions in the online seminar. We employed a classification system developed in our earlier work (Jones, Scanlon and Blake, 2000;Blake and Scanlon, 2012) and validated and extended it using the rating scheme by Meier et al for assessing the quality of computer supported collaboration processes (see Meier et al 2007 andRummel et al 2011). In these analyses we took the whole message as the unit of analysis as suggested by Rourke et al (2001) as it combines the flexibility of the thematic unit with the reliable identification of attributes of a syntactical unit.…”

Section: Collaborative Processesmentioning

confidence: 99%

Design for Collaboration

Blake

Scanlon

2013

JIME

View full text Add to dashboard Cite

Online learning environments offer new opportunities for learning and over the last decade or so a variety of online learning environments have been developed by researchers to facilitate collaborative learning among students. In this paper we will present a case study of a successful collaborative learning design. This involves a near synchronous online seminar where students work in small groups to produce a report that examines media coverage of controversial science, using archives of television news reports. We will analyze the activity's task design features by employing a framework of collaboration enabling design approach proposed by Kirschner et al (2004). We will start with an analysis of the collaborative processes and interaction among participants in this online activity. Then we will examine the features of the task used in this learning environment with respect to the interaction design ideas proposed by Kirschner et al. They suggest that the use of appropriately designed and implemented educational, social and technological affordances is the foundation for stimulating, motivating and maintaining collaboration among learners. We use the framework to identify factors contributing to the success of the activity.

show abstract

Analyzing Collaborative Interactions Across Domains and Settings: An Adaptable Rating Scheme

Cited by 17 publications

References 25 publications

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA

Quantifying Collaboration Quality in Face-to-Face Classroom Settings Using MMLA

Design for Collaboration

Contact Info

Product

Resources

About