Measurement Properties of Two Innovative Item Formats in a Computer-Based Test

Abstract

Many innovative item formats have been proposed over the past decade, but little empirical research has been conducted on their measurement properties. This study examines the reliability, efficiency, and construct validity of two innovative item formats-the figural response (FR) and constructed response (CR) formats used in a K-12 computerized science test. The item response theory (IRT) information function and confirmatory factor analysis (CFA) were employed to address the research questions. It was found that the FR items were similar to the multiple-choice (MC) items in providing information and efficiency, whereas the CR items provided noticeably more information than the MC items but tended to provide less information per minute. The CFA suggested that the innovative formats and the MC format measure similar constructs. Innovations in computerized item formats are reviewed, and the merits as well as challenges of implementing the innovative formats are discussed.Recent years have seen rapid growth in the use of computers in educational and licensure/certification testing. Many tests are now administered on computer, and software has been developed to facilitate all aspects of test development. On paper-and-pencil tests, multiple-choice (MC) items are the dominant testing format (Scalise & Gifford, 2008). Now armed with new technologies, test users and developers have shown increased interest in employing innovative item formats.Many innovative item formats have emerged over the past decade, but little empirical research has been conducted on their measurement properties. The purpose of this study is to empirically examine the reliability, efficiency, and Correspondence should be addressed to Lei Wan,

Predictive Adaptive Test with Selective Weighted Bayesian Through Questions and Answers Patterns to Measure Student Competency Levels

“…Jeong [5] compared the scores of Korean students on computer-based and paper-based test versions. Wan and Henly [6] in their study of CAT found that figural response items are similar to multiple-choice items in providing information and efficiency. Meanwhile, response items made offer more information than multiple-choice items but tend to provide less information per minute.…”

Are Smileys Valid Answers? Survey Data Quality with Innovative Item Formats

“…IIFs are being used extensively in questionnaires, especially within the use of tests. The higher use of IIFs over time may in part be due to pressure from politicians to use innovation to get better tests, since some national learning tests have been criticized for not having high enough validity and reliability (Crabtree 2016), but also due to higher popularity of their use among respondents (Wan & Henly 2012). Upon reflections over 100 years of survey research Krosnick said "These are exciting times for survey research."…”

“…The name suggests innovation is an aspect that shapes how item types are evolved. Other researchers have later used the same term (Wan & Henly 2012) and some researchers have used the term Technology-Enhanced Items, (TEI, Scalise & Gifford 2006, Crabtree 2016). Zheng (2011) mentions several innovative item types: highlight, re-order or fill in blanks in text in different variations.…”

Psychometric properties of technology-enhanced item formats

“…However, with the educational advancements that computer technology has enabled, test users and developers have shown increased interest in employing TE item formats. Many item formats using technological enhancements have been proposed over the past decade, but little empirical research has been conducted on their measurement properties (Wan and Henly, 2012;Serici and Zensky, 2006). In an effort to address the lack of research in this area, this study examines TE item formats, specifically the construct validity, information, and efficiency of these formats when used to supplement traditional MC items on educational assessments.…”

“…Past studies also indicated that test score reliability could be lower for tests composed of constructed response questions given fixed testing time (Lukhele et al, 1994;Thissen, et al, 1994;Wainer & Thissen, 1993). For other TE item types, measurement research is limited or nonexistent (Wan and Henly, 2012). Jodoin (2003) analyzed innovative item formats for IRT information to determine the measurement information and efficiency of the new formats.…”