Modeling Key Differences in Underrepresented Students' Interactions with an Online STEM Course

Bosch, Nigel; Crues, R. Wes; Henricks, Genevieve M.; Perry, Michelle; Angrave, Lawrence; Shaik, Najmuddin; Bhat, Suma; Anderson, Carolyn J.

doi:10.1145/3183654.3183681

Cited by 11 publications

(14 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, the realtime accessibility of behavioral clickstream data can be used to develop automatic feedback and intervention modules within the LMS. For example, researchers have built early detection systems for dropout or poor course performance, which can help instructors allocate their attention to the most at-risk students (Baker, Lindrum, Lindrum, and Perkowski, 2015;Bosch et al 2018;Lykourentzou, Giannoukos, Nikolopoulos, Mpardis, and Loumos, 2009;Whitehill, Williams, Lopez, Coleman, and Reich, 2015). Students can also be provided with adaptive guidance in real-time by, for instance, suggesting collaboration partners (Brusilovsky, 2003;Caprotti, 2017).…”

Section: Clickstream Data and Its Use In Higher Education Researchmentioning

confidence: 99%

The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: opening the black box of learning processes

Baker

Park

et al. 2020

Int J Educ Technol High Educ

View full text Add to dashboard Cite

Student clickstream data-time-stamped records of click events in online courses-can provide fine-grained information about student learning. Such data enable researchers and instructors to collect information at scale about how each student navigates through and interacts with online education resources, potentially enabling objective and rich insight into the learning experience beyond self-reports and intermittent assessments. Yet, analyses of these data often require advanced analytic techniques, as they only provide a partial and noisy record of students' actions. Consequently, these data are not always accessible or useful for course instructors and administrators. In this paper, we provide an overview of the use of clickstream data to define and identify behavioral patterns that are related to student learning outcomes. Through discussions of four studies, we provide examples of the complexities and particular considerations of using these data to examine student self-regulated learning.

show abstract

Section: Clickstream Data and Its Use In Higher Education Researchmentioning

confidence: 99%

The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: opening the black box of learning processes

Baker

Park

et al. 2020

Int J Educ Technol High Educ

View full text Add to dashboard Cite

show abstract

“…Marginalization and longstanding underrepresentation in STEM often leave UR-STEM students experiencing additional barriers not experienced by their non-UR-STEM peers, such as a lack of social support, negative stereotypes, lower academic self-efficacy, and a lack of sense of belonging, which may work together to impact their engagement and achievement [2,22,31,44]. Past research on UR-STEM students in online courses has shown that the online environment increases accessibility to STEM programs for these students [19,70] and there is evidence that UR-STEM students engage with online learning platforms differently from their non-UR-STEM peers [3]. Unfortunately, some UR-STEM students (in this case, first-generation college students) may be lacking the SRL skills necessary for success in online courses [68], and other UR-STEM students (women) have been found to be more likely to perform poorly in or drop out of online vs. in-person STEM courses (compared to men) [71].…”

Section: Engagement In Online Coursesmentioning

confidence: 99%

A Social Network Analysis of Online Engagement for College Students Traditionally Underrepresented in STEM

Williams-Dobosz

Azevedo

Jeng

et al. 2021

LAK21: 11th International Learning Analytics and Knowledge Conference

Self Cite

View full text Add to dashboard Cite

Little is known about the online learning behaviors of students traditionally underrepresented in STEM fields (i.e., UR-STEM students), as well as how those behaviors impact important learning outcomes. The present study examined the relationship between online discussion forum engagement and success for UR-STEM and non-UR-STEM students, using the Community of Inquiry (CoI) model as our theoretical framework. Social network analysis and nested regression models were used to explore how three different measures of forum engagement-1) total number of posts written, 2) number of help-seeking posts written and replied to, and 3) level of connectivity-were related to improvement (i.e., relative performance gains) for 70 undergraduate students enrolled in an online introductory STEM course. We found a significant positive relationship between help-seeking and improvement and nonsignificant effects of general posting and connectivity; these results held for UR-STEM and non-UR-STEM students alike. Our findings suggest that online help-seeking has benefits for course improvement beyond what can be predicted by posting alone and that one need not be well connected in a class network to achieve positive learning outcomes. Finally, UR-STEM students demonstrated greater grade improvement than their non-UR-STEM counterparts, which suggests that the online environment has the potential to combat barriers to success that disproportionately affect underrepresented students.

show abstract

“…There are situations where this may not be the case, however. For example, Figure 1 illustrates varying the decision threshold of a logistic regression model that predicts whether a university student is enrolled as a science major or not (Bosch et al, 2018). In this case, kappa and precision can be improved by predicting fewer positive cases (46.2%) than the data class proportions suggest (68.3%).…”

Section: Introductionmentioning

confidence: 99%

Metrics for Discrete Student Models: Chance Levels, Comparisons, and Use Cases

Bosch

Paquette

2018

Learning Analytics

Self Cite

View full text Add to dashboard Cite

Metrics including Cohen's kappa, precision, recall, and F1 are common measures of performance for models of discrete student states, such as a student's affect or behaviour. This study examined discrete model metrics for previously published student model examples to identify situations where metrics provided differing perspectives on model performance. Simulated models also systematically showed the effects of imbalanced class distributions in both data and predictions, in terms of the values of metrics and the chance levels (values obtained by making random predictions) for those metrics. Random chance level for F1 was also established and evaluated. Results for example student models showed that overprediction of the class of interest (positive class) was relatively common. Chance-level F1 was inflated by over-prediction; conversely, maximum possible values for F1 and kappa were negatively impacted by overprediction of the positive class. Additionally, normalization methods for F1 relative to chance are discussed and compared to kappa, demonstrating an equivalence between kappa and normalized F1. Finally, implications of results for choice of metrics are discussed in the context of common student modelling goals, such as avoiding false negatives for student states that are negatively related to learning. Notes for Practice• Previous research has shown that choice of metric plays a key role in training and evaluation of student models, focusing primarily on metrics intended for models that produce probabilistic predictions of student outcome variables• Imbalances in labelled data are quite common in student modelling tasks, and have been shown to impact metrics used for machine-learned student models• This paper explores the impact that predicted class proportions and data class proportions have on discrete model metrics including Cohen's kappa, precision, recall, and F1, and formulates a random-chance level F1 measurement that is adjusted for imbalances• Results on real-world student models and simulated models show that best practices include reporting multiple metrics for discrete student models, and comparing F1 scores to the appropriate chance level to avoid over-or under-estimating model performance

show abstract

Modeling Key Differences in Underrepresented Students' Interactions with an Online STEM Course

Cited by 11 publications

References 11 publications

The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: opening the black box of learning processes

The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: opening the black box of learning processes

A Social Network Analysis of Online Engagement for College Students Traditionally Underrepresented in STEM

Metrics for Discrete Student Models: Chance Levels, Comparisons, and Use Cases

Contact Info

Product

Resources

About