Scaling Performance Assessments: Strategies for Managing Local Item Dependence

Yen, Wendy M.

doi:10.1111/j.1745-3984.1993.tb00423.x

Cited by 543 publications

(574 citation statements)

References 15 publications

Supporting

Mentioning

535

Contrasting

Unclassified

Order By: Relevance

“…This implies that there are no dependencies between the items except those attributed to the latent trait (Marais & Andrich, 2008;Yen, 1993). Violations of local independence may be indicated by residual correlations and multidimensionality (Ip, 2010).…”

Section: Discussionmentioning

confidence: 99%

Time for a new approach to prepare future teachers for educational technology use: Its meaning and measurement

Tondeur

Braak

Siddiq

et al. 2016

Computers & Education

214

120

View full text Add to dashboard Cite

The main objective of this study is to develop a self-report instrument to measure preservice teachers' perceptions of the extent to which they experience the necessary support and training in order to integrate technology into classroom activities. The questionnaire items of this instrument were drawn up on the basis of a synthesis of 19 qualitative studies (Authors et al., 2012) and were reviewed by experts in the field. In order to study its reliability and aspects of validity, data were collected and analysed consisting of a sample of 688 pre-service teachers in Flanders (Belgium). The resulting scale showed highly satisfactory psychometric properties. Item response theory revealed a good fit of the measurement to a Rating Scale Model for 22 out of 24 items. The results also indicate that the items differ in their degree of difficulty. It seems that helping pre-service teachers to design ICT-rich lessons and providing adequate feedback can be considered more challenging for teacher training institutions.Recommendations are given regarding how the new scale can be useful for both teacher training institutions and schools in developing approaches to equip pre-service teachers with the competencies needed to integrate technology in teaching and learning processes.

show abstract

Section: Discussionmentioning

confidence: 99%

Time for a new approach to prepare future teachers for educational technology use: Its meaning and measurement

Tondeur

Braak

Siddiq

et al. 2016

Computers & Education

214

120

View full text Add to dashboard Cite

show abstract

“…A critical assumption is that the items are conditionally independent. This assumption is unlikely to be true if a number of items are related to one another through reference to a common stem, for such structures foster local dependence (Yen, 1993). Reading comprehension and listening comprehension testlets, as previously described, are examples of structures that are likely to yield local dependence.…”

Section: + + +mentioning

confidence: 97%

How Reliable Is the Toefl Test?

Wainer

Lukhele

1997

ETS Research Report Series

View full text Add to dashboard Cite

EDUCATIONAL TESTING SERVICE, ETS, the ETS logo, GRE, TOEFL, and the TOEFL logo are registered trademarks of Educational Testing Service. Abstract!We estimated the reliability of scores on four forms of the Test of English as a Foreign Language (TOEFL@) using a hybridIRT model. Very little difference between their overall reliability was found when the testlet items were assumed to be independent and their dependence was modeled. A larger difference in reliability was found when test sections were analyzed individually. Then we found as much as a 40 percent overestimate in reading comprehension testlets, with the longer testlets of the newest form of the TOEFL test showing the most local dependence. The listening comprehension testlets exhibited much less local dependence. We also found that the test was unidimensional enough for the use of univariate IRT to be efficacious, and that the reading comprehension testlets showed essentially no differential functioning by sex. This paper will be appearing in the journal Educational and Psychological Measurement and all references to it should be to that source.1 This research was supported by the TOEFL Policy Council and we are pleased to acknowledge its help. In addition, the data we analyzed were made available to us by Linda Tang who also advised us about some of their idiosyncratic characteristics; the data files themselves were prepared by Stephen Laue. We are grateful to Dan Eignor, Jacqueline Ross, and Mary Schedl for their careful reading of an earlier draft that helped us to correct some errors and allowed us to say better what we meant; also for most of the semicolons. Last, we would like to thank David Thissen for his always helpful advice. Of course none of these people or organizations should be held responsible for any errors herein, we claim total responsibility for those ourselves. This work was accomplished while the second author was an ETS post-doctoral fellow; his current address is 5804 Zone 5, Pimville, Johannesburg 1808, Republic of South Africa.The Test of English as a Foreign Language (TOEFL~) was developed in 1963 by the National Council on the Testing of English as a Foreign Language. The Council was formed through the cooperative effort of more than 30 public and private organizations concerned with testing the English proficiency of nonnative speakers of the language applying for admission to institutions in the United States. In 1965, Educational Testing Service (ETSifI) and the College Board assumed joint responsibility for the program. In 1973, a cooperative arrangement for the operation of the program was entered into by ETS, the College Board, and the Graduate Record Examinations (GRE ifI ) Board. The membership of the College Board is composed of schools, colleges, school systems, and educational associations; GRE Board members are associated with graduate education.ETS administers the TOEFL program under the general direction of a Policy Council that was established by, and is affiliated with, the sponsoring organizations. Members of...

show abstract

“…For example, the skills or competencies that students demonstrate during a PBA are rated by one or more raters, usually resulting in a categorization of competency mastery as, for instance, insufficient/sufficient. However, it has often been demonstrated that PBAs score low on generalizability, reliability, and standardization, and that rater effects can influence students' scores (Kane 1990;Linn et al 1991;Ruiz-Primo et al 1993;Shavelson et al 1993;Yen 1993;Messick 1995;Dekker and Sanders 2008). MBPA might be used as an alternative and improved approach to the measurement of vocational skills.…”

Section: Introductionmentioning

confidence: 99%

A framework for designing and developing multimedia-based performance assessment in vocational education

Klerk¹,

Veldkamp

Eggen³

2017

Education Tech Research Dev

View full text Add to dashboard Cite

The development of any assessment should be an iterative and careful process. Ideally, this process is guided by a well-defined framework (see for ), but such a framework is not always available when the instrument to be developed is new or innovative. Frameworks for the development of traditional computer-based tests have been published and experimented with since the late 1990s, by which time CBT had already existed for more than a decade. In an earlier empirical pilot study, we described a new type of assessment for Dutch vocational education, called multimedia-based performance assessment (MBPA) (self-revealing reference 2014). This CBT uses multiple media formats and interactive tasks to measure skills that are currently measured by performancebased assessment. In conducting that pilot study, deficits in the existing literature made it difficult to ground all developmental steps in sound scientific theory. To remedy those deficits, this article presents and validates a framework for the design and development of MBPA, combining a search of the relevant literature from several subfields of educational assessment and consultation with assessment experts. The framework unites assessment development and multimedia development theory, focus solely on vocational education, -017-9559-5 and answers the call for a framework from the scientific community. The first step in validating the prototype framework involved five semi-structured interviews with Dutch assessment and multimedia experts to produce a final version of the framework. Second, the pilot MBPA was reconstructed in accordance with this finalized framework, resulting in an improved MBPA and demonstrating that the proposed framework is a useful and applicable tool for the design and development of MBPA in vocational education.example

show abstract

Scaling Performance Assessments: Strategies for Managing Local Item Dependence

Cited by 543 publications

References 15 publications

Time for a new approach to prepare future teachers for educational technology use: Its meaning and measurement

Time for a new approach to prepare future teachers for educational technology use: Its meaning and measurement

How Reliable Is the Toefl Test?

A framework for designing and developing multimedia-based performance assessment in vocational education

Contact Info

Product

Resources

About