Judgment accuracy in experienced versus student teachers: Assessing essays in English as a foreign language

Jansen, Thorben; Vögelin, Cristina; Machts, Nils; Keller, Stefan; Köller, Olaf; Möller, Jens

doi:10.1016/j.tate.2020.103216

Cited by 26 publications

(17 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Achievement-irrelevant information was not directly related to performance but was often taken into account by teachers when assessing student performance, as shown in some previous studies (e.g., Ready and Wright 2011;Ritts et al 1992;Schrader and Helmke 2001). This judgment error has also been reported in other studies that used the SI with preservice, trainee, and in-service teachers in English and mathematics (Jansen et al 2019(Jansen et al , 2021Kaiser et al 2015). To the best of our knowledge, our results are the first to provide an indication that the halo effect also occurs in trainee teachers of biology.…”

Section: Discussionsupporting

confidence: 77%

“…Educational psychological research has described numerous judgment errors made by teachers that contribute significantly to the distortion of judgment accuracy. The knowledge about judgment accuracy (facet of PK) is therefore necessary in order for teachers to make accurate judgments (e.g., Jansen et al 2019Jansen et al , 2021Kaiser et al 2015;Schrader 2006;Vögelin et al 2019). Studies have already shown that teachers have problems in assessing relevant performance without including previously shown performance, which should actually be assessed independently.…”

Section: Rqmentioning

confidence: 99%

“…Kaiser et al (2015) were able to show that the achievement-irrelevant information German grade, intelligence, and gender (female) had a positive effect on the assessment of mathematics performance and thus biased the accuracy of judgments. Based on the same theoretical framework, a study by Jansen et al (2019Jansen et al ( , 2021) that used the SI with pre-service teachers investigated, among other things, the extent to which students' gender or an immigrant background (achievement-irrelevant information) influenced the assessment of students' English essays. Experimental variation in achievement-relevant (essay quality) and achievement-irrelevant (gender, immigrant background) information showed no effect on the assessment accuracy of the English essays.…”

Section: The Student Inventory (Si)mentioning

confidence: 99%

“…We used the SI on the diagnostic competence of preservice, trainee, and in-service teachers, which was developed by Kaiser et al (2015) and used to investigate educational psychological research questions (Jansen et al 2019(Jansen et al , 2021. We adapted it for our purposes to measure the generic knowledge about judgment accuracy (i.e., diagnosis of achievement-relevant information; facet of PK) and, simultaneously, biology-specific facets of professional knowledge such as the knowledge about the core ideas and principles of evolution (i.e., diagnosis of scientific correctness; facet of CK) and the knowledge about student understanding (i.e., diagnosis of specific misconception categories; facet of PCK).…”

Section: The Student Inventory (Si)mentioning

confidence: 99%

See 3 more Smart Citations

Measuring biology trainee teachers’ professional knowledge about evolution—introducing the Student Inventory

et al. 2021

Self Cite

View full text Add to dashboard Cite

Background To teach evolution efficiently teachers must be able to diagnose their students’ ideas and understanding of the phylogeny of organisms. This encompasses different facets of content-specific professional knowledge, that is, knowledge about core ideas and theories, as well as knowledge about respective misconceptions. However, as findings from the field of psychology have shown, diagnostic activities comprise a further facet, namely, teachers’ judgment accuracy. This refers to the question of whether achievement-irrelevant information about the student influences teachers’ diagnoses. Against this background we conducted a study (1) to assess trainee teachers’ abilities to diagnose (a) the scientific correctness of students’ written answers, (b) students’ misconceptions about evolution, and (2) to investigate the interplay of evolution specific and generic facets of professional knowledge during the diagnosis. For this purpose, we applied a digital instrument, the Student Inventory (SI). Using this instrument, the trainee teachers (N = 27) first diagnosed written answers (N = 6) from virtual students regarding their scientific correctness and regarding students’ misconceptions about the natural selection of the peppered moth. Second, to test for judgment accuracy, the trainee teachers received—via the SI—achievement-irrelevant information about each virtual student, that is, the previous result of a multiple-choice questionnaire about evolution, before diagnosing the written answers. Results The trainee teachers were able to distinguish between scientifically correct (90.8%) and scientifically incorrect (91.7%) written answers. Trainee teachers faced problems when diagnosing specific misconceptions categories. Anthropomorphic misconceptions were diagnosed significantly more often (61.1%) than teleological misconceptions (27.8%). The achievement-irrelevant information influenced the trainee teachers’ assessment of written answers (F [1,26] = 5.94, p < .022, η2 = .186) as they scored the written answers higher if the performance in the questionnaire was good and vice versa. Conclusion The findings indicate that the diagnosis is easier or more difficult depending on the particular misconception category. However, the findings also reveal that, besides the evolution-specific facets of professional knowledge, generic facets interrelate with the quality of the diagnosis result. We conclude from these findings that an integration of evolution-specific and generic knowledge into the education of biology teachers is critical.

show abstract

Section: Discussionsupporting

confidence: 77%

Section: Rqmentioning

confidence: 99%

Section: The Student Inventory (Si)mentioning

confidence: 99%

Section: The Student Inventory (Si)mentioning

confidence: 99%

See 2 more Smart Citations

Measuring biology trainee teachers’ professional knowledge about evolution—introducing the Student Inventory

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Participants assessed each text on the six-level holistic scale of the National Assessment of Educational Progress (Driscoll et al, 2010). This study further employed genre-specific analytic rating scales (for an overview, see Jansen, Vögelin, Machts, Keller, Köller, & Möller, 2021). These seven scales were based on the 6 + 1 trait model (Culham, 2003) as well as the Test in English for Educational Purposes (TEEP) (Weir, 1988), and we adjusted them to address genre-specific characteristics of argumentative essays (Hyland, 1990;Zemach & Stafford-Yilmaz, 2008).…”

Section: Text Assessmentmentioning

confidence: 99%

Don't Just Judge the Spelling! The Influence of Spelling on Assessing Second-Language Student Essays

Jansen

Vögelin

Machts

et al. 2021

FLR

Self Cite

View full text Add to dashboard Cite

When judging subject-specific aspects of students’ texts, teachers should assess various characteristics, e.g., spelling and content, independently of one another since these characteristics are indicators of different skills. Independent judgments enable teachers to adapt their classroom instruction according to students’ skills. It is still unclear how well teachers meet this challenge and which intervention could be helpful to them. In Study 1, N = 51 pre-service teachers assessed four authentic English as a Second Language (ESL) essays with different overall text qualities and different qualities of spelling using holistic and analytic rating scales. Results showed a negative influence of the experimentally manipulated spelling errors on the judgment of almost all textual characteristics. In Study 2, an experimental prompt was used to reduce this judgment error. Participants who were made aware of the judgment error caused by spelling errors formed their judgments in a less biased way, indicating a reduction of bias. The determinants of the observed effects and their practical implications are discussed.

show abstract

Digitale Schreibförderung im Bildungskontext: Ein systematisches Review

et al. 2023

View full text Add to dashboard Cite

Judgment accuracy in experienced versus student teachers: Assessing essays in English as a foreign language

Cited by 26 publications

References 46 publications

Measuring biology trainee teachers’ professional knowledge about evolution—introducing the Student Inventory

Measuring biology trainee teachers’ professional knowledge about evolution—introducing the Student Inventory

Don't Just Judge the Spelling! The Influence of Spelling on Assessing Second-Language Student Essays

Digitale Schreibförderung im Bildungskontext: Ein systematisches Review

Contact Info

Product

Resources

About