The neglected situation: assessment performance and interaction in context

Maddox, Bryan

doi:10.1080/0969594x.2015.1026246

Cited by 13 publications

(15 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The application of these perspectives is evidence of a trend away from routine procedures of validation practice (Zumbo and Chan, 2014), towards more eclectic and improvised study of assessment performance that integrates sources of environmental feedback from the testing situation (author names removed, 2017). This supports a more sensitive distinction between ecological noise and signal to inform decisions about what counts as relevant sources of variance (Maddox, 2015).…”

supporting

confidence: 52%

“…These are observable though the study of every-day talk. This paper follow's Goffman's lead to study rea-life testing situations as distinctive social occasions (see McNamara, 1997;Maddox, 2015). The application of these perspectives is evidence of a trend away from routine procedures of validation practice (Zumbo and Chan, 2014), towards more eclectic and improvised study of assessment performance that integrates sources of environmental feedback from the testing situation (author names removed, 2017).…”

mentioning

confidence: 99%

See 1 more Smart Citation

Talk and Gesture as Process Data

Maddox

2017

Measurement: Interdisciplinary Research and Perspectives

Self Cite

View full text Add to dashboard Cite

This paper examines the topic of talk and gesture in computer-based tests, and what it can tell us about assessment performance. The paper describes a micro-analytic approach, and uses video-ethnographic methods to investigate data that is not captured by conventional assessment scores or by computer-generated log files. The paper presents transcripts from assessment events in the OECD Adult Skills Survey (PIAAC) in Slovenia. The transcripts describe a three-way interaction between interviewer, respondent and computer. They enable a comparison of the roles of the computer and interviewer in the management of assessment quality and engagement. The paper demonstrates the ways that interaction in testing situations influences test performance, and how respondents make cognitive-affective decisions about their engagement. In conclusion, the paper considers how process data on talk and gesture can inform validation practice

show abstract

supporting

confidence: 52%

mentioning

confidence: 99%

Talk and Gesture as Process Data

Maddox

2017

Measurement: Interdisciplinary Research and Perspectives

Self Cite

View full text Add to dashboard Cite

show abstract

“…Unlike conventional think aloud protocols, these verbal commentaries were unprompted and took place during the actual assessment events (for reviews of think aloud protocols, see Downloaded by [University of Otago] at 06:33 28 July 2015 Willis, 2004;Greene, Robertson, & Costa, 2011). The frequent verbal interactions between testers, respondents, and bystanders provided insights into the problemsolving strategies of respondents and any difficulties they had with the test format and content (see Maddox, 2015a).…”

Section: Methodsmentioning

confidence: 99%

“…Responses to three test items (Timed Parking, the Gas Gauge, and the Mongolian Camel) are examined using a combination of ethnographic transcripts and differential item functioning (DIF) analysis. This interdisciplinary analysis builds on an in-depth ethnographic account of assessment events in the Mongolian assessment-in-situ observations of individual assessments as they occurred in the LAMP assessment (Maddox, 2014(Maddox, , 2015a. The potentials of this mixed-method and interdisciplinary approach is the focus of the article.…”

Section: Introductionmentioning

confidence: 98%

An Anthropologist Among the Psychometricians: Assessment Events, Ethnography, and Differential Item Functioning in the Mongolian Gobi

Maddox

Zumbo

Tay-Lim

et al. 2015

International Journal of Testing

Self Cite

View full text Add to dashboard Cite

This article explores the potential for ethnographic observations to inform the analysis of test item performance. In 2010, a standardized, large-scale adult literacy assessment took place in Mongolia as part of the United Nations Educational, Scientific and Cultural Organization Literacy Assessment and Monitoring Programme (LAMP). In a novel form of interdisciplinary collaboration an ethnographer worked closely with psychometric researchers to investigate the sources and explanations of differential item functioning and test item performance. The research involved detailed ethnographic observations of literacy assessment events. The results illustrate the potential for ethnography to provide insights into testing situations, group and test item performance, and to combine with psychometric analysis in cross-cultural assessment

show abstract

“…Bryan Maddox (2015) has focused upon the experience of taking a large-scale assessment test in Mongolia. Bryan Maddox (2015) has focused upon the experience of taking a large-scale assessment test in Mongolia.…”

mentioning

confidence: 99%

On test development and accuracy in self-assessment

Hopfenbeck

2015

Assessment in Education: Principles, Policy & Practice

View full text Add to dashboard Cite

EDITORIALOn test development and accuracy in self-assessmentThe opening article by Jin and Hamp-Lyons (2015) in this regular issue reports on an empirical investigation of the need for a new test for the assessment of English for professional purposes in China. As the authors point out, it is not often the need for a new test is investigated empirically; tests are usually introduced by educational policy-makers or suggested by commercial organisations for the test's potential market value and profits. With reference to Shohamy (2001), Jin and Hamp-Lyons remind us that the voices of stakeholders such as the learners, teachers and potential users of the test are seldom heard. The article therefore offers an important contribution to the field of language testing by reporting the initial phases of the development of a new language assessment. In the first phase, an online survey was conducted collecting data on stakeholders' perceptions of the necessity of the test (N = 327), representing three stakeholder groups: (1) university students, (2) university graduates employed by joint ventures or multinational companies and (3) university teachers of advanced English courses. In the second phase, the researchers analysed five English language tests, (Business English Certificate -Higher, Test of English for International Communication (TOEIC), CET-6, Test for English Majors and Public English Test System) to see whether existing tests would fit the need for the new test, or inform the development of the new proposed test. The first two tests are developed for an international marked, while the final three are developed for China. Finally, the researchers reviewed six language frameworks (Foreign Service Institute, or Interagency Language Table, Australian Second Language Proficiency Ratings (ASLPR), Proficiency Guidelines of American Council on the Teaching of Foreign Languages, Can-Do statements of Association of Language Testers in Europe, CEFR, and Canadian Language Benchmarks) to evaluate whether any of them could serve as guidelines for the design of the new assessment, particularly the test's specifications. Data analysis of the survey revealed some discrepancies among the stakeholder groups on the perceptions of the needs of communication in English in China, but all groups agreed on the need to develop a test of EPP for Chinese learners of English at an advanced level. The analysis of the five tests further showed that none of the Chinese tests on the marked were sophisticated enough for the purpose of assessing English proficiency at an advanced level for multinational professional workplaces in China. The review of the existing frameworks also showed that the level descriptions were too general for any practical use for the test developers, and more related to vocational purposes than professional purposes. The authors therefore argued for the need to develop a new test in China with real value for potential employers. The study is a timely reminder of the importance for researchers and test-designers to inves...

show abstract

The neglected situation: assessment performance and interaction in context

Cited by 13 publications

References 35 publications

Talk and Gesture as Process Data

Talk and Gesture as Process Data

An Anthropologist Among the Psychometricians: Assessment Events, Ethnography, and Differential Item Functioning in the Mongolian Gobi

On test development and accuracy in self-assessment

Contact Info

Product

Resources

About