Assessing cognitive presence in online inquiry‐based discussion through text classification and epistemic network analysis

Abstract

Providing coaching to participants in inquiry‐based online discussions contributes to developing cognitive presence (CP) and higher‐order thinking. However, a primary issue limiting quality and timely coaching is instructors' lack of tools to efficiently identify CP phases in massive discussion transcripts and effectively assess learners' cognitive development. This study examined a computational approach integrating text mining and co‐occurrence analysis for assessing CP and cognitive development in online discussions based on the community of inquiry (CoI) framework. First, text classifiers trained on different language models were evaluated for identifying and coding the CP phases. Second, epistemic network analysis (ENA) was employed to model cognitive patterns reflected by co‐occurrences between the coding elements. Results indicated that text classifiers trained on the state‐of‐the‐art language model Bidirectional Encoder Representations from Transformers (BERT) can address the efficiency issue in coding CP phases in discussion transcripts and obtain substantial agreements (Cohen's k = 0.76) with humans, which outperformed other baseline classifiers. Furthermore, compared to traditional quantitative content analysis, ENA can effectively model the temporal characteristics of online discourse and detect fine‐grained cognitive patterns. Overall, the findings suggest a feasible path for applying learning analytics to tracking learning progression and informing theory‐based assessments. What is already known about this topic Cognitive presence is an important construct describing the progression of thinking in online inquiry‐based discussions. Most studies used self‐report instruments or quantitative content analysis to measure and assess cognitive presence. More efficient and effective approaches were needed by instructors to support assessment of cognitive development and determine coaching strategies. What this paper adds An integrated computational approach for the developmental and formative assessment of cognitive presence was proposed and evaluated. A BERT‐based text classification model could efficiently code massive transcripts and achieve substantial agreements with human coders. Epistemic network analysis effectively revealed the process of cognitive development and identified representative discussion patterns and behaviours. Implications for practice and/or policy The proposed approach can considerably reduce the pressure on instructors, enabling them to focus on quality coaching and feedback. Compared to frequencies of individual codes, the connective features between codes carry more insights for assessing cognitive patterns. Learners in a discussion group play different roles and produce diverse paths of cognitive development.

Learning analytics and assessment: Emerging research trends, promises and future opportunities

“…Sun and Theussen (2022) applied social network analysis to dynamically assess the development of students' negotiation skills and demonstrated that complexity of negotiation skills was positively related to negotiation outcome in a collaborative simulation game. Ba et al (2022) developed text classifiers to detect phases in cognitive presence and then used epistemic network analysis to identify cognitive patterns in student online discussion transcripts. Temporal and fine-grained analytics approaches presented in this study may be a promising venue towards advancing dynamic and theory-aligned assessments of computer-supported collaborative learning.…”

Using Epistemic Network Analysis to Explore Primary School Students’ Computational Thinking in Pair Programming Learning

“…ENA has great potential to understand the development of students’ CT in collaborative programming. ENA can effectively detect differences in connections between codes with similar frequencies compared with automated content analysis (Ba et al, 2022) and quantitative content analysis (Csanadi et al, 2018). It also goes beyond traditional static quantitative analysis to capture progressive relationships between different variables, such as cognition and interaction (Guo et al, 2023).…”

“…Epistemic Network Analysis (ENA) is a discourse analysis method. Due to its ability to reveal the interaction, support, and temporal process in the collaboration (Zhang et al, 2022), ENA has been used in many studies to understand the development trajectory of learners’ ideas in scientific creativity tasks (Sun et al, 2022), higher-order thinking (Ba et al, 2022), metacognitive patterns (Wu et al, 2020), social-cognitive engagement (Ouyang et al, 2022), and even to compare metacognitive differences between low- and high-scoring groups (Wu et al, 2020; Zhang et al, 2019). ENA can model the connections between features of student dialogue by quantifying their co-occurrences in the dialogue, and thus produce a weighted network of co-occurrences and visualizations linked with each data analytical component.…”

“…Hence, through the visual network, it is possible to find the development trajectory in students’ CT skills. Sun et al (2022) discussed students’ development trajectory of divergent thinking, and Ba et al (2022) explored students’ development process of higher-order thinking in online inquiry-based discussion via ENA. Thus, this study aimed to apply ENA to analyze the process trajectory of CT among primary school learners in PP, as well as to compare the difference between the low- and high-scoring students’ CT.…”

“…However, Scratch could not visualize the formation process of students’ CT (Stewart et al, 2021). ENA, as a discourse analysis tool, can present the formation process of thinking (Ba et al, 2022; Sun et al, 2022), which provides a perfect opportunity for visualizing the formation process of CT. Thus, this study aimed to apply ENA to compare the difference between low- and high-scoring students’ CT in PP learning.…”