Modeling Holistic Marks With Analytic Rubrics

Tomas, Carmen; Whitt, Emma; Lavelle-Hill, Rosa; Severn, Katie E.

doi:10.3389/feduc.2019.00089

Cited by 16 publications

(19 citation statements)

References 41 publications

(118 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We also found that unpacking of analytic bins can help determine if the item elicits critical components of the holistic rubric and if the item elicits all 5 levels of the LP. This is similar to a finding reported by Tomas et al (2019) that having humans apply both holistic and analytic codes helped define some of the criteria underlying the holistic scoring. We found that we were able to achieve good performing models with both ML platforms tested, although there is some evidence that CRC performed better than LightSide on some items using analytic composite scores, this was not true for all items or true for using holistic scores.…”

Section: Discussionsupporting

confidence: 88%

“…However, this increase in model accuracy was not uniformly achieved across all items and took significant effort and time to achieve, mainly in rubric design and additional coding time. Harsch and Martin (2013) and Tomas et al (2019) argue that holistic and analytic approaches can be combined to offer productive outcomes for both research and practice. One possible extension of this is to investigate the combination of both holistic and analytic coding to generate one improved ML model.…”

Section: Discussionmentioning

confidence: 99%

“…Holistic rubrics are defined in this experiment as being polytomous with each scoring-level being mutually exclusive to the others and which captures a unique and often complex set of response characteristics. Capturing complex content interactions is claimed to be easier with holistic rubrics than analytic rubrics (Tomas et al 2019). Holistic rubrics provide a comprehensive evaluation of a response and are thought of as easy to use (Jescovitch et al 2019b;Yune et al 2018).…”

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

“…A key consideration in the application of either type of rubric is inter-rater reliability (IRR), or how closely scores assigned by multiple raters to a single response agree. Comparisons of studies employing holistic or analytic rubrics approaches to human coding are summarized by Tomas et al (2019) and Brookhart (2018), and find there is little evidence to support one approach as universally "better" than the other for human coding. This is an important consideration for developing ML models, as the reliability of human coding in the training dataset is a key requirement for achieving high performing scoring models (Liu et al 2016).…”

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

“…Models for analytic bins trained with low frequency positive cases but with less lexical diversity, or fewer unique words used, performed better than models trained with higher frequency positive cases but exhibiting very diverse language. Tomas et al (2019) identified CR assessments as divergent, or exhibiting rich lexical diversity, in that criteria can be met by a broad range of words and phrases within student individual responses. For example, in RMP2, bin 11, or "there are more K+ channels (than Na+ channels)," predicted 21 positive cases (human identified 32 positive cases) with a HC κ = 0.79.…”

Section: Analytic Human-computer Inter-rater Reliability Unpackedmentioning

confidence: 99%

See 4 more Smart Citations

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

et al. 2020

View full text Add to dashboard Cite

We systematically compared two coding approaches to generate training datasets for machine learning (ML): (i) a holistic approach based on learning progression levels and (ii) a dichotomous, analytic approach of multiple concepts in student reasoning, deconstructed from holistic rubrics. We evaluated four constructed response assessment items for undergraduate physiology, each targeting five levels of a developing flux learning progression in an ion context. Human-coded datasets were used to train two ML models: (i) an 8-classification algorithm ensemble implemented in the Constructed Response Classifier (CRC), and (ii) a single classification algorithm implemented in LightSide Researcher’s Workbench. Human coding agreement on approximately 700 student responses per item was high for both approaches with Cohen’s kappas ranging from 0.75 to 0.87 on holistic scoring and from 0.78 to 0.89 on analytic composite scoring. ML model performance varied across items and rubric type. For two items, training sets from both coding approaches produced similarly accurate ML models, with differences in Cohen’s kappa between machine and human scores of 0.002 and 0.041. For the other items, ML models trained with analytic coded responses and used for a composite score, achieved better performance as compared to using holistic scores for training, with increases in Cohen’s kappa of 0.043 and 0.117. These items used a more complex scenario involving movement of two ions. It may be that analytic coding is beneficial to unpacking this additional complexity.

show abstract

Section: Discussionsupporting

confidence: 88%

Section: Discussionmentioning

confidence: 99%

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

Section: Analytic Human-computer Inter-rater Reliability Unpackedmentioning

confidence: 99%

See 3 more Smart Citations

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

et al. 2020

View full text Add to dashboard Cite

show abstract

Introductory biology undergraduate students' mixed ideas about genetic information flow

Uhl

Sripathi

Saldanha

et al. 2020

Biochem Molecular Bio Educ

View full text Add to dashboard Cite

The core concept of genetic information flow was identified in recent calls to improve undergraduate biology education. Previous work shows that students have difficulty differentiating between the three processes of the Central Dogma (CD; replication, transcription, and translation). We built upon this work by developing and applying an analytic coding rubric to 1050 student written responses to a three-question item about the CD. Each response was previously coded only for correctness using a holistic rubric. Our rubric captures subtleties of student conceptual understanding of each process that previous work has not yet captured at a large scale. Regardless of holistic correctness scores, student responses included five or six distinct ideas. By analyzing common co-occurring rubric categories in student responses, we found a common pair representing two normative ideas about the molecules produced by each CD process. By applying analytic coding to student responses preinstruction and postinstruction, we found student thinking about the processes involved was most prone to change. The combined strengths of analytic and holistic rubrics allow us to reveal mixed ideas about the CD processes and provide a detailed picture of which conceptual ideas students draw upon when explaining each CD process.

show abstract

Assessing the quality of science teachers' lesson plans: Evaluation and application of a novel instrument

Großmann,

Krüger

2023

Science Education

View full text Add to dashboard Cite

Lesson planning is a core part of teachers' professional competence. Written lesson plans play a significant role in science teacher education as a preparation for demonstration lessons during the final teacher certification exam. However, the few existing scoring rubrics on lesson plans are not particularly theoretically sound and are barely tested for the validity of score interpretations. In response to the demand for transparent and applicable criteria, we developed the rubric to assess science lesson plans (RALP) to assess science teachers' lesson plan quality. We employed a mixed‐methods approach: First, we present multiple sources of validity evidence (based on test content, internal structure, relations to other variables, and consequences of testing) as mainly quantitative indicators for the quality of the RALP. Based upon that, we applied the RALP to lesson plans written by preservice and trainee science teachers (N = 100) and provided a qualitative analysis of six cases to illustrate common patterns in these lesson plans. Results indicate that teacher educators consider the RALP criteria (N = 24) relevant and objectively applicable. Correlation analyses of the scores and two teacher educators' holistic quality assessment of all lesson plans provide convincing evidence that the RALP can discriminate lesson plan quality levels. Moreover, comparisons between preservice science teachers and trainee science teachers reveal that trainee teachers score significantly higher than preservice teachers, indicating that the RALP is sensitive to differences in teaching and planning experience. The application and in‐depth analysis of three criteria of the RALP illustrate these differences in levels of planning quality. We discuss possible applications of the RALP in science teacher education and research in science teaching.

show abstract

Modeling Holistic Marks With Analytic Rubrics

Cited by 16 publications

References 41 publications

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

Introductory biology undergraduate students' mixed ideas about genetic information flow

Assessing the quality of science teachers' lesson plans: Evaluation and application of a novel instrument

Contact Info

Product

Resources

About