ITEM ANALYSIS FOR TEACHER‐MADE MASTERY TESTS¹

Abstract

Various item selection techniques are compared on resultant criterionreferenced reliability and validity. Techniques compared include three nominal criterion-referenced methods, a traditional point biserial selection, teacher selection, and random selection.Eighteen volunteer junior and senior high school teachers supplied behavioral objectives and item pools ranging from 26 to 40 items. Each teacher obtained reponses from four classes. Pairs of tests of various length were developed by each item selection method. Estimates of test reliability and validity were obtained using responses independent of the test construction sample. Resultant reliability and validity estimates were compared across item selection techniques. Two of the criterion-referenced item selection methods resulted in consistently higher observed validity. However, the small magnitude of improvement over teacher or random selection raises a question as to whether the benefit warrants the necessary extra effort on the part of the classroom teacher.Glaser and Nitko (1971) define a criterion-referenced measure (CRM) as "'one that is deliberately constructed so as to yield measurements that are directly interpretable in terms of specified performance standards." A CRM differs from the more familiar norm-referenced measure (NRM) in that interpretation of the latter depends on performance relative to other individuals rather than on an a priori criterion.Popham and Husek (1969) point out the limitations of norm-referenced measurement techniques for the development and evaluation of CRMs. Their major criticism is the dependence of norm-referenced measurement on score variability; e.g., most normreferenced techniques of estimating test reliability and validity are directly related to score variability. However, a "good" CRM administered after adequate instruction may result in very low score variation. Since this test would fail to satisfy usual normreferenced standards of reliability and validity, new criterion-referenced concepts of reliability and validity are necessary.Popham and Husek also question the interpretation of traditional norm-referenced item discrimination indices for use in criterion-referenced item selection. Typical norm-referenced item discrimination indices relate item score to total test score. An item with a low discrimination index on a NRM is suspect. However, in criterionreferenced measurement, if an item accurately measures an essential behavior and all examinees respond correctly (or incorrectly), the item cannot be faulted.A mastery test is a CRM used in formative evaluation. The mastery test is designed to measure the essential behaviors associated with each unit of instruction. Given following instruction, the mastery test allows for a dichotomous decision regarding the student's mastery status and identifies the deficiencies of the student below criterion. The student demonstrating mastery would go on to the next unit of instruction while tThis paper is based on the author's Ph.D. dissertation submitted to the faculty of...

Criterion-Referenced Testing: Review, Evaluation, and Extension

“…For the two test validity indices, however, Crehan 's index and the Cox and Vargas measure produced tests which showed significant and consistently higher test validities over the other four i tem measures. Wi th these results , Crehan (1974) …”

“…A second view of CR measurement (e.g., Crehan, 1974;Glaser & Cox, 1968) considers these two topics to be intima tely related due to the emphasis of both on performance. In practice, CR measurement often consists largely or completely of performance testing.…”

“…Positively discriminating i tems might be used to locate areas requiring additiona l or modified instruction so that all students can be brought to the mastery level--a goa l of the instructiona l program designed under the CR testing concept. Negatively discriminating items (i.e., items answered correctly more often by the low total test scoring students ) may serve to identify a need for revision either in the items or in the focus of instruction and instructional materi al (Hambleton & Gorth , 1971 ;Gorth & Hambleton , 1972;Popham & Husek , 1969 Cox and Vargas (1966), Ivens (1970), Rahml ow , Mathews and Jung (1970), Hambleton and north (1971), Hsu (1971), Popham (1971) , Crehan (1974), and Haladyna (1974).…”

A Latent Trait Look at Pretest-Posttest Validation of Criterion-referenced Test Items

“…Most of these studies also try to detect clusters of coefficients and use correlation techniques for this purpose without realizing that most clusters can be expected a priori because of spurious correlations (the coefficients are defied on the same proportions from Table I). Crehan (1974) evaluates a number of pretest-posttest coefficients in view of their possible contributions to improving mastery decisions, and Smith (Note 1) does the same not using empirical but simulated data. The fact that he chooses the Rasch model to simulate the data and therewith represents items having equal discriminating power might account for his not finding any trend.…”

Criterion Referenced Testing: Review, Evaluation, and Extension.

“…- Crehan (1974) Test reliability was estimated as the proportion of agreement in overall test grades (pass or fail) on two parallel test taken by the same group of students. An agreement, for example , was two passing grades.…”