The Critical Thinking Analytic Rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments

Saxton, Emily; Belanger, Secret; Becker, William

doi:10.1016/j.asw.2012.07.002

Cited by 47 publications

(33 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Finally, it is important to point out that this study was limited to analyzing inter-rater reliability. However, as reported by Saxton et al (2012), it may be useful to focus on inter-and intra-rater reliability and examine both types of rater behavior as a path to improving assessment processes. Therefore, future research may focus on comparing the differences found between different raters as well as those found between scores awarded by the same rater.…”

Section: Discussion Of Resultsmentioning

confidence: 99%

“…The results of this study allowed the researchers to conclude that the use of assessment tools, such as scoring rubrics, is not enough to improve rater reliability (González & Roux, 2013;Saxton et al 2012;Weigle, 1994), and therefore these results may have important implications for teaching practice. First, it is important to consider that rubrics are tools that teachers may use to facilitate their assessment of writing.…”

Section: Conclusion and Teaching Implicationsmentioning

confidence: 99%

“…For instance, raters differ when they judge students' writing ability depending on their linguistic (Shi, 2001) and educational background (Mendelsohn & Cumming, 1987). Their judgment also depends on the type of scoring scale used and the interpretation that raters give to the scale (Bacha, 2001;Barkaoui, 2007;Knoch, 2009;Saxton, Belanger, & Becker, 2012;Attali, Lewis & Steier, 2012). This study describes the analytical grade variability found between the grades that 11 Mexican university EFL teachers awarded to five sample papers written by university students.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Assessing EFL university students' writing: A study of score reliability

González

Trejo

Roux

2017

REDIE

View full text Add to dashboard Cite

The assessment of English as a Foreign Language (EFL) writing is a complex activity that is subject to human judgment, which makes it difficult to achieve a fair, accurate and reliable assessment of student writing (Pearson, 2004: 117;Hamp-Lyons, 2003). This study reports on the variability that exists between the analytical grades that 11 Mexican EFL university teachers awarded to five writing samples. It describes the raters' views on writing assessment and their use of analytical scoring rubrics. Data obtained from the grades awarded to each paper, and a background questionnaire, suggested that great variability was found between grades, and raters differed in their levels of leniency and severity, suggesting that having similar backgrounds and using the same rubric are not enough to ensure rater reliability. Participants' perceptions were found to be similar in terms of the use of analytical rubrics.Palabras: EFL writing, EFL writing assessment, Rater reliability, Analytical scoring rubric. ResumenLa evaluación de la escritura en inglés como lengua extranjera (EFL) es un proceso que depende del juicio humano, por lo que es difícil de obtener evaluaciones justas, acertadas y confiables (Pearson, 2004, p. 117;Hamp-Lyons, 2003). Este estudio reporta la variabilidad existente entre las calificaciones analíticas que 11 docentes Mexicanos universitarios de EFL proporcionaron a cinco trabajos escritos. Describe las percepciones de los participantes en torno a la evaluación de la escritura y el uso de las rúbricas de evaluación. Los datos obtenidos de las calificaciones de cada trabajo y de un cuestionario escrito revelaron que existe gran variedad entre las calificaciones proporcionadas y que los evaluadores difieren en sus niveles de exigencia, sugiriendo así que antecedentes homogéneos y el uso de una misma rúbrica 1 The authors presented preliminary results of this study at the 2014 TESL Canada Conference at the University of Regina, Saskatchewan, Canada, May 8-10, 2014 Assessing EFL university students' writing: a study of score reliability Palabras clave: Inglés como lengua extranjera, Evaluación de la escritura, Confiabilidad, Rúbrica de evaluación.

show abstract

Section: Discussion Of Resultsmentioning

confidence: 99%

Section: Conclusion and Teaching Implicationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assessing EFL university students' writing: A study of score reliability

González

Trejo

Roux

2017

REDIE

View full text Add to dashboard Cite

show abstract

“…En esta línea, también Rekalde y Buján (2014) llegaron a la conclusión de que las rúbricas permiten valorar lo que el alumnado ha conseguido y lo que le falta por conseguir. Asimismo, los estudiantes concretan como ventaja del instrumento que el mismo permite ocupar el lugar del docente, en tanto que, facilita anotar muestras de trabajo de una manera sistemática (Saxton, Belanger & Becker, 2012).…”

Section: Discusión Y Conclusionesunclassified

Opinión del alumnado sobre el empleo de rúbricas en la universidad

2017

View full text Add to dashboard Cite

RESUMENEl actual modelo educativo de formación en competencias requiere que todos los elementos del currículo giren en torno al desarrollo de las mismas por parte del alumnado. También la evaluación debe encaminarse a comprobar el nivel de logro de las competencias requeridas. Para ello se necesita variedad de instrumentos de recogida de información que pongan de manifiesto la diversidad de competencias a adquirir. Una de las herramientas más utilizadas en la actualidad para este fin son las denominadas rúbricas. Por ello, en esta investigación nos propusimos analizar el grado de adecuación de dichas herramientas de evaluación desde la percepción del alumnado universitario. Mediante la aplicación del método descriptivo no experimental, se aplicó un cuestionario de elaboración propia a 440 estudiantes de Pedagogía. Los resultados indican buena validez y fiabilidad de dicho instrumento, así como que las rúbricas poseen altas ventajas frente a los inconvenientes que pueden conllevar.Palabras clave: evaluación de competencias, rúbricas, educación superior.ABSTRACT The current educational model of training requires that all elements of the curriculum revolve around students' development of competences. Also, the assessment should be directed to check the level of achievement of the required competences. For this purpose, a variety of data collection instruments are necessary in order to reveal the diversity of competences to acquire. Rubrics are one of the main tools used for this purpose. So in this investigation we decided to analyze the grade of adequacy of these assessment tools from the perception of university students. By applying a non-experimental descriptive method, we applied a self-development questionnaire to 440 students of Pedagogy. The results indicate good validity and reliability of the instrument and stablish the rubrics have high advantages over the possible disadvantages associated.

show abstract

“…What is deemed as an acceptable level of inter-rater reliability, across this literature review, depended on the type of assessment; for example, consider formative vs. highstakes summative, and the importance of a particular level of inter-rater reliability in each case. In examining a few specific examples of similar studies with a small number (2-7) of raters focusing on the assessment of capstone experiences, communication, critical thinking and portfolio development, a combination of percentage agreement and a correlation coefficient was used to measure inter-rater reliability [1,7,9,12,14,15].…”

Section: Literature On Inter-rater Reliabilitymentioning

confidence: 99%