Test architecture, test retrofit

Abstract

Just like buildings, tests are designed and built for specific purposes, people, and uses. However, both buildings and tests grow and change over time as the needs of their users change. Sometimes, they are also both used for purposes other than those intended in the original designs. This paper explores architecture as a metaphor for language test development. Firstly, it describes test purpose and use, and how this affects test design. Secondly, it describes and illustrates the layers of test architecture and design. Thirdly, it discusses the concept of test retrofit, which is the process of altering the test after it has been put into operational use. We argue that there are two types of test retrofit: an upgrade and a change. Each type of retrofit implies changes to layers of the test architecture which must be articulated for a validity argument to be constructed and evaluated. As is true in architecture, we argue that a failure to be explicit about retrofit seriously limits validity claims and clouds issues surrounding the intended effect of the test upon users.

Bayesian Networks in Educational Assessment

“…The domain modeling process helps clarify purposes and highlight trade-offs that are involved. Moreover, a domain model helps a design team think through what needs to be changed about an assessment to meet a new purpose (Fulcher and Davidson 2009). …”

15. Assessment in educational settings

“…Validation is concerned with the gathering of as much evidence as possible, which would support or refute the inferences that are made about test takers based on their performance on the test, and the decisions that are made about learners based on their test scores. According to Fulcher and Davidson (2009) the intended score meaning should be explicitly and carefully linked to test design, as otherwise it is not possible to relate the users' interpretation of the score to the decisions that they take on the basis of the score. Another issue that may undermine the appropriateness of tests is the lack of need for justifying the validity of a test among decision makers who are predominantly not language assessment experts.…”

Effective rating scale development for speaking tests: Performance decision trees

“…We are now in a position to outline the content and nature of a scoring method for complex service encounters which, being directly linked to performance and task, would form a critical part of the architecture of a complete service encounter test (Fulcher and Davidson, 2009 Figure 1. This is a prototype scoring system for tasks, as well as providing a clearer picture of what competencies and skills are required to successfully engage in complex service encounters.…”