Validation of Automated Scoring for a Formative Assessment that Employs Scientific Argumentation

Mao, Liyang; Liu, Ou Lydia; Roohr, Katrina Crotts; Belur, Vinetha; Mulholland, M. Robert; Lee, Hee‐Sun; Pallant, Amy

doi:10.1080/10627197.2018.1427570

Cited by 57 publications

(42 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These scoring systems identify important conceptual components in student responses, which then can be used to classify responses. Other efforts have applied automated scoring systems to assess scientific practices in middle school science, such as argumentation (Mao et al 2018;Haudek et al 2019). These efforts identified key components of the practice of argumentation (e.g., claim) via automated scoring.…”

Section: Machine Learning Of Constructed Response Assessmentsmentioning

confidence: 99%

“…Other previous work has used multi-leveled coding schemes to try to characterize the quality of complete student responses (Liu et al 2016;Mao et al 2018;Wiley et al 2017). Such holistic coding schemes may be developed based on "correctness" of an explanation; however, not all such schemes may be closely tied to an underlying framework.…”

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

See 1 more Smart Citation

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

et al. 2020

View full text Add to dashboard Cite

We systematically compared two coding approaches to generate training datasets for machine learning (ML): (i) a holistic approach based on learning progression levels and (ii) a dichotomous, analytic approach of multiple concepts in student reasoning, deconstructed from holistic rubrics. We evaluated four constructed response assessment items for undergraduate physiology, each targeting five levels of a developing flux learning progression in an ion context. Human-coded datasets were used to train two ML models: (i) an 8-classification algorithm ensemble implemented in the Constructed Response Classifier (CRC), and (ii) a single classification algorithm implemented in LightSide Researcher’s Workbench. Human coding agreement on approximately 700 student responses per item was high for both approaches with Cohen’s kappas ranging from 0.75 to 0.87 on holistic scoring and from 0.78 to 0.89 on analytic composite scoring. ML model performance varied across items and rubric type. For two items, training sets from both coding approaches produced similarly accurate ML models, with differences in Cohen’s kappa between machine and human scores of 0.002 and 0.041. For the other items, ML models trained with analytic coded responses and used for a composite score, achieved better performance as compared to using holistic scores for training, with increases in Cohen’s kappa of 0.043 and 0.117. These items used a more complex scenario involving movement of two ions. It may be that analytic coding is beneficial to unpacking this additional complexity.

show abstract

Section: Machine Learning Of Constructed Response Assessmentsmentioning

confidence: 99%

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

et al. 2020

View full text Add to dashboard Cite

show abstract

“…B. dem Kodierleitfaden) verglichen werden, um auf diese Weise den automatisierten Auswertungsprozess inhaltlich zu prüfen (Williamson et al 2012;Yang et al 2002). Über Korrelationsanalysen lässt sich darüber hinaus ermitteln, ob die Kodierung durch die Algorithmen stärker durch die Antwortlänge beeinflusst wird als die menschliche Kodierung, was als Mangel für Validität betrachtet wird (Mao et al 2018).…”

Section: üBerprüfung Der Güte Des Maschinellen Lernensunclassified

“…Hierzu wird die Übereinstimmung zwischen menschlicher und computerbasierter Kodierung sowie der Prozess der automatisierten Kodierung untersucht (Yang et al 2002 Eine hohe Übereinstimmung der durch die Algorithmen besonders stark gewichteten Attribute mit den Ausführungen im Kodierleitfaden (F3) wird als Evidenz für eine valide Interpretation der durch die Algorithmen vorgenommenen Kodierung betrachtet (Williamson et al 2012). Ferner liefern Korrelationsanalysen zwischen den durch die Algorithmen kodierten Niveaus mit der externen Variable "Antwortlänge" (F4) Hinweise darauf, inwieweit die Algorithmen von einem Oberflächenmerkmal beeinflusst werden (Mao et al 2018 Tab. 2 und 3).…”

Section: Zielsetzung Und Fragestellungenunclassified

Maschinelles Lernen mit Aussagen zur Modellkompetenz

Krüger

Krell

2020

ZfDN

View full text Add to dashboard Cite

Zusammenfassung Verfahren des maschinellen Lernens können dazu beitragen, Aussagen in Aufgaben im offenen Format in großen Stichproben zu analysieren. Am Beispiel von Aussagen von Biologielehrkräften, Biologie-Lehramtsstudierenden und Fachdidaktiker*innen zu den fünf Teilkompetenzen von Modellkompetenz (NTraining = 456; NKlassifikation = 260) wird die Qualität maschinellen Lernens mit vier Algorithmen (naïve Bayes, logistic regression, support vector machines und decision trees) untersucht. Evidenz für die Validität der Interpretation der Kodierungen einzelner Algorithmen liegt mit zufriedenstellender bis guter Übereinstimmung zwischen menschlicher und computerbasierter Kodierung beim Training (345–607 Aussagen je nach Teilkompetenz) vor, bei der Klassifikation (157–260 Aussagen je nach Teilkompetenz) reduziert sich dies auf eine moderate Übereinstimmung. Positive Korrelationen zwischen dem kodierten Niveau und dem externen Kriterium Antwortlänge weisen darauf hin, dass die Kodierung mit naïve Bayes keine gültigen Ergebnisse liefert. Bedeutsame Attribute, die die Algorithmen bei der Klassifikation nutzen, entsprechen relevanten Begriffen der Niveaufestlegungen im zugrunde liegenden Kodierleitfaden. Abschließend wird diskutiert, inwieweit maschinelles Lernen mit den eingesetzten Algorithmen bei Aussagen zur Modellkompetenz die Qualität einer menschlichen Kodierung erreicht und damit für Zweitkodierungen oder in Vermittlungssituationen genutzt werden könnte.

show abstract

“…The argument is increasingly recognized as a fundamental and important intellectual skill to be learned (Crowell & Kuhn 2014;Scheuer, Pinkwart, & McLaren 2010). Arguments become a core part that should be applied in the classroom (Mao, Liu, Roohr, Belur, Mulholland, Lee, & Pallant 2018). Many international educational curricula that requires students to be able to participate in the argument, debate, and decision-making on science issues (Dawson & Carson 2017).…”

Section: Introductionmentioning

confidence: 99%

The Technology Pedagogy Knowledge (TPK) Teacher Using Worksheet 3D Pageflip Professional for Promoting Argumentation Skills High-Schools Students in Physics Learning

Oktasari

Kuswanto

Ismet

et al. 2018

jpppf

View full text Add to dashboard Cite

The demands of 21st-century skills highlighted the importance of cooperation between teachers and students to achieve the expected skillthe Pedagogical knowledge of teachers in teaching and technology into things that need to be developed. The pedagogic ability of teachers in the use of technology can be a potential to improve students' abilities argument. This study aims to develop students' ability argumentation through pedagogical knowledge of teachers in using technology. The technology used is the result of the development of student worksheets 3D PageFlip Impulse and Momentum topics. The purpose of this study to look at the ability of students through the application of scientific argumentation (Technological Pedagogic Knowledge) TPK teachers in the classroom. This research is descriptive research. The data collection capability student argumentation performed by administering a written test. The subjects were students of class X at SMAN 1 Yogyakarta Prambanan. Rate scientific argumentation ability of students refers to Toulmin Argumentation Pattern (TAP). The results showed that the complexity of argumentation ability of students still at low level. These results suggest that the ability of the student arguments need to be improved. makes backing achieved a score of 2.3, with the category enough. In general, the complexity of argumentation ability of students is at a low level (claims with a single ground).

show abstract

Validation of Automated Scoring for a Formative Assessment that Employs Scientific Argumentation

Cited by 57 publications

References 23 publications

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

Maschinelles Lernen mit Aussagen zur Modellkompetenz

The Technology Pedagogy Knowledge (TPK) Teacher Using Worksheet 3D Pageflip Professional for Promoting Argumentation Skills High-Schools Students in Physics Learning

Contact Info

Product

Resources

About