Educational Testing and Test‐form Factors

Abstract

Evidence is here brought together suggesting that educational and other psychological tests are appreciably dependent not only on the abilities that they are supposed to measure, but on the form and material of the tests and their items, sets and attitudes in the testees, and various conditions of testing. The factors that emerge when tests are inter‐correlated represent to some extent these formal characteristics rather than desired intellectual functions. In particular this applies to current educational tests which, for sound educational reasons, avoid simpler, more factual, items and prefer complex comprehension‐type or problem items.Part II shows that, while tests of the same objectives employing different forms tend to give discrepant results (e.g., essay and new‐type), tests in the same form which are aimed at different school subjects or different intellectual functions inter‐correlate very highly. This detracts from their validity, especially in counseling or differentiating abilities along different lines. For many purposes the simpler tests show superior validity, and it is doubtful how far the more complex ones do bring in the ‘higher’ intellectual functions at which they are aimed.Part III argues that, in like manner, most of the cognitive factors established by psychologists show little relevance to educational or other external criteria, possibly because they represent factors within paper‐and‐pencil tests rather than genuine factors of intellect.Part IV. A first essential is to map out more precisely the extent and nature of form influences, so that, in so far as they prove to be important, account can be taken of them in test construction. A number of types of predictive measures are discussed which might add more to the validity of educational selection, particularly at college level, than do complex forms of reading tests. These include: better use of essay examinations and school grades, listening comprehension tests, speed and error scores, indices of study habits, tests based on learning situations, and tests of rational thinking, flexibility and creativity. When tests are used primarily for evaluation rather than prediction, investigations should be guided by the conception of intrinsic or construct validity, which – unlike logical or face validity – involves the collection of empirical evidence. Some of the more promising approaches to such problems are through factorial studies of educational tests along with tests of psychological factors and ‘real‐life’ criteria, and experimental studies of group differences and of the gains brought about by different types of teaching.

Convergent and discriminant validation by the multitrait-multimethod matrix.

“…Studies of individual differences among laboratory animals resulted in the recognition of "apparatus factors," usually more dominant than psychologi-VALIDATION BY THE MULTITRAIT-MULT1 METHOD MATRIX 85 cal process factors (Tryon, 1942). For paper-and-pencil tests, methods variance has been noted under such terms as "test-form factors" (Vernon: 1957(Vernon: , 1958 and "response sets" (Cronbach: 1946(Cronbach: , 1950Lorge, 1937). Cronbach has stated the point particularly clearly: "The assumption is generally made .…”

On Test‐Wiseness and Some Related Constructs

“…A similar meaning of TW has been given by Diamond & Evans (1972), Ebel & Damrin (1960), Moore, Schutz & Baker (1966), Oakland & Weilert (1972), Stanley (1971), Thorndike (1951) and Vernon (1958). There is a corresponding meaning in what Loevinger (1967) calls 'secular trends' and what Guilford terms 'faking-good', 'sophistication' and 'falsification'.…”

“…In the systematization of possible sources of variation in knowledge and ability measures by Stanley (1971; see also Thorndike 1951and Vernon 1958, TW is treated partly as lasting and general and partly as temporary and general. With reference to this systematization Thorndike writes (1951):…”

Objective Measurement of Personality: Cautiousness and Intolerance of Ambiguity1

“…Successive drawings in a figure tab item, hmfever, represented fairly constant increases in information. TIle figure tab score also had signifi- (Cattell, 1957, po 238), to which it bears some similarity, and of couxse the contribution of test-form variance due to a connnon tab format mUl;lt be evaluated (Vernon, 1958 for by content ability. The correlation between the tab scores on the two tests was .34, and the coefficient tended to increase slightly…”