Differential item functioning analysis of the Vanderbilt Expertise Test for cars

Abstract

The Vanderbilt Expertise Test for cars (VETcar) is a test of visual learning for contemporary car models. We used item response theory to assess the VETcar and in particular used differential item functioning (DIF) analysis to ask if the test functions the same way in laboratory versus online settings and for different groups based on age and gender. An exploratory factor analysis found evidence of multidimensionality in the VETcar, although a single dimension was deemed sufficient to capture the recognition ability measured by the test. We selected a unidimensional three-parameter logistic item response model to examine item characteristics and subject abilities. The VETcar had satisfactory internal consistency. A substantial number of items showed DIF at a medium effect size for test setting and for age group, whereas gender DIF was negligible. Because online subjects were on average older than those tested in the lab, we focused on the age groups to conduct a multigroup item response theory analysis. This revealed that most items on the test favored the younger group. DIF could be more the rule than the exception when measuring performance with familiar object categories, therefore posing a challenge for the measurement of either domain-general visual abilities or category-specific knowledge.

Age-related differential item functioning in tests of face and car recognition ability

“…Cho et al (2015) found that, at the test level, the CFMT showed very little DIF across gender groups, across individuals tested online or in the lab, and across younger versus older individuals. In contrast, while Lee et al (2015) found no evidence of test-level gender DIF in the VET-Car, they found a significant number of DIF items between age groups (,27 or ! 27 years).…”

“…The present work investigates age-related differential item functioning (DIF) in two new matching tasks designed to measure face recognition and car recognition abilities. This work is motivated by earlier reports of more age-related DIF in the measurement of car recognition ability (Lee, Cho, McGugin, Van Gulick, & Gauthier, 2015) than in measurement of face recognition ability (Cho et al, 2015). The presence of considerable age bias in the measurement of car recognition ability is a concern because many questions in high-level vision use a comparison between different categories, very often face and nonface objects, with cars often serving as the sole nonface category (Richler, Wilmer, & Gauthier, 2017;Shakeshaft & Plomin, 2015).…”

“…Several tests of nonface object recognition have been created using the same format (Dennett et al, 2012;McGugin, Richler, Herzmann, Speegle, & Gauthier, 2012;Richler et al, in press;. Two recent studies employed IRT methods to test the assumption of invariance across groups in the CFMT (Cho et al, 2015) and in a similar test of car recognition, the Vanderbilt Expertise Test for cars (VET-Car; Lee et al, 2015). To evaluate bias within tests, IRT DIF can identify differences in item parameters or item response functions after controlling for differences in levels of performance on the latent trait.…”

Gender differences in recognition of toy faces suggest a contribution of experience

“…However, the male advantage in performance for cars is also not significant when study time is not regressed out but only the AMT subjects are included in the ANOVA ( p = 0.22). This may reflect the age difference between AMT and lab subjects (see Table 1), especially since prior work has reported an influence of age on the measurement of car recognition ability (Lee, Cho, McGugin, Van Gulick, & Gauthier, 2015) but not faces (Cho et al, 2015). Whatever the reason, this should not be taken to suggest that the gender difference in performance with cars is due to differences in study time.…”

Validation of the Vanderbilt Holistic Face Processing Test

“…Out of a pool of 1000 participants who performed the VET-car on Amazon Mechanical Turk (see Lee et al, 2015), we sent 384 invitations to male participants 1 stating they were eligible for new tasks and would receive a bonus if they completed them all. These tasks included the composite task with top judgments (completed by 195 participants), the composite task with bottom judgments (completed by 181 participants), the VET-bird (completed by 180 participants, but these data were not analyzed because several trials were repeated by mistake), and the VHPT-F (completed by 174 participants).…”