Why are some GCSE examination questions harder to mark accurately than others? Using Kelly's Repertory Grid technique to identify relevant question features

Suto, Irenka; Nádas, Rita

doi:10.1080/02671520801945925

Cited by 13 publications

(13 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Differences in the properties of the rating scale: The formulation and conceptualisation of rating scales have varied between the different research studies that have used KRG. Some studies have used a 5-point scale (e.g., Suto & Na´das, 2009), whilst some have found five points to be limiting and have used a 7-point scale instead (e.g., Gray, 1999).…”

Section: Resultsmentioning

confidence: 98%

“…If a research project seeks to build knowledge that can be generalised beyond an individual participant, it might be desirable to interview a number of participants. In such cases, it would be good practice to run individual interviews in sequence with the same researcher(s) leading each session (for a detailed account, see Suto & Na´das, 2009. Otherwise, interviews could be run in parallel with the researchers using clearly designed protocols to ensure that acceptable levels of between-participant consistency of approach are attained (see, e.g., Greatorex et al, 2011).…”

Section: Issues Around the Potential Misapplication Of The Krg Methodmentioning

confidence: 97%

“…It is also best practice for the researcher to use a written interview protocol (see Suto & Na´das 2009). This helps to structure the interview and reminds the researcher to keep checking the meaning of elicited constructs with the participant.…”

Section: Issues Around the Potential Misapplication Of The Krg Methodmentioning

confidence: 99%

“…Note: This example uses actual constructs from Suto and Na´das (2008). A detailed account of the process is found in Suto and Na´das (2009).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A review of the uses of the Kelly's Repertory Grid method in educational assessment and comparability research studies

Johnson

Nádas

2012

Educational Research and Evaluation

Self Cite

View full text Add to dashboard Cite

Comparability between different educational qualifications is an important issue within policy discourse in the UK. In this context, the comparability of qualification demands has been explored through the use of expert human judgement. The involvement of human judgement in estimating assessment demands has consequences for methodology. This review considered the ways that the Kelly's Repertory Grid (KRG) technique has been used to compare assessment demands over recent years. The review involved the identification and analysis of research documents as well as consideration of the original theory on which the KRG method was founded. This article describes how the technique has been adapted as it has shifted from its original psychotherapeutic context to be used in educational assessment and comparability studies. The review also explored possible disadvantages that stem from these adaptations, leading to recommendations to ensure the validity of findings when using the technique in educational assessment studies.

show abstract

Section: Resultsmentioning

confidence: 98%

Section: Issues Around the Potential Misapplication Of The Krg Methodmentioning

confidence: 97%

Section: Issues Around the Potential Misapplication Of The Krg Methodmentioning

confidence: 99%

“…Note: This example uses actual constructs from Suto and Na´das (2008). A detailed account of the process is found in Suto and Na´das (2009).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A review of the uses of the Kelly's Repertory Grid method in educational assessment and comparability research studies

Johnson

Nádas

2012

Educational Research and Evaluation

Self Cite

View full text Add to dashboard Cite

show abstract

“…For instance, researchers in England have used KRG to demonstrate how examination questions can be preclassified systematically in terms of the cognitive strategies and therefore the rater expertise needed to score them accurately (Suto & Nádas, 2009, 2010). This task is necessary because in many of England's popular public qualifications, a variety of question types are used within a single examination paper, generating examinee responses of diverse formats and lengths.…”

Section: Kelly's Repertory Gridmentioning

confidence: 99%

A Critical Review of Some Qualitative Research Methods Used to Explore Rater Cognition

Suto¹

2012

Educational Measurement

View full text Add to dashboard Cite

Internationally, many assessment systems rely predominantly on human raters to score examinations. Arguably, this facilitates the assessment of multiple sophisticated educational constructs, strengthening assessment validity. It can introduce subjectivity into the scoring process, however, engendering threats to accuracy. The present objectives are to examine some key qualitative data collection methods used internationally to research this potential trade‐off, and to consider some theoretical contexts within which the methods are usable. Self‐report methods such as Kelly's Repertory Grid, think aloud, stimulated recall, and the NASA task load index have yielded important insights into the competencies needed for scoring expertise, as well as the sequences of mental activity that scoring typically involves. Examples of new data and of recent studies are used to illustrate these methods’ strengths and weaknesses. This investigation has significance for assessment designers, developers and administrators. It may inform decisions on the methods’ applicability in American and other rater cognition research contexts.

show abstract

A Historical Survey of Research Regarding Constructed-Response Formats

Bejar

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

This chapter chronicles ETS research and development contributions related to the use of constructed-response item formats. 1 The use of constructed responses in testing dates back to imperial China, where tests were used in the selection of civil servants. However, in the United States, the multiple-choice format became dominant during the twentieth century, following its invention and use by the SAT ® examinations created by the College Board in 1926. When ETS was created in 1947, post-secondary admissions testing was largely based on tests consisting of multiplechoice items. However, from the start, there were two camps at ETS: those who believed that multiple-choice tests were sufficiently adequate for the purpose of assessing "verbal" skills and those who believed that "direct" forms of assessment requiring written responses had a role to play. For constructed-response formats to regain a foothold in American education several hurdles would need to be overcome. Research at ETS was instrumental in overcoming those hurdles.The first hurdle was that of reliability, specifically the perennial issue of low interrater agreement, which plagued the acceptance of constructed-response formats for most of the twentieth century. The second hurdle was broadening the conception of validity to encompass more than predictive considerations, a process that began with the introduction of construct validity by Cronbach and Meehl (1955). Samuel Messick at ETS played a crucial role in this process by making construct validity relevant to educational tests. An inflexion point in the process of reincorporating constructed-response formats more widely in educational tests was marked 1 Constructed responses to a prompt or question can range in scope and complexity. Perhaps the most common constructed response is the written essay. However, short written responses to questions are also considered to be constructed, as are spoken answers in response to a prompt, mathematical responses (equations, plotted functions, etc.), computer programs, and graphical responses such as architectural designs.

show abstract

Why are some GCSE examination questions harder to mark accurately than others? Using Kelly's Repertory Grid technique to identify relevant question features

Cited by 13 publications

References 21 publications

A review of the uses of the Kelly's Repertory Grid method in educational assessment and comparability research studies

A review of the uses of the Kelly's Repertory Grid method in educational assessment and comparability research studies

A Critical Review of Some Qualitative Research Methods Used to Explore Rater Cognition

A Historical Survey of Research Regarding Constructed-Response Formats

Contact Info

Product

Resources

About