What Makes Domain Knowledge Difficult? Word Usage Frequency From SUBTLEX and dlexDB Explains Knowledge Item Difficulty

Abstract

The quality of tests in psychological and assessment educational is of great scholarly and public interest. Item difficulty models are vital to generating test result interpretations based on evidence. A major determining factor of item difficulty in knowledge tests is the opportunity to learn about the facts and concepts in question. Knowledge is mainly conveyed through language. Exposure to language associated with facts and concepts might be an indicator of the opportunity to learn. Thus, we hypothesize that item difficulty in knowledge tests should be related to the probability of exposure to the item content in everyday life and/or academic settings and therefore also to word frequency. Results from a study with 99 political knowledge test items administrated to N = 250 German 7th (age: 11 – 14 years) and 10th (age: 15 – 18 years) graders showed that word frequencies in everyday settings (SUBTLEX-DE) explain variance in item difficulty, while word frequencies in academic settings (dlexDB) alone do not. However, both types of word frequency combined explain a considerable amount of the variance in item difficulty. Items with words that are relatively more frequent in everyday life compared to academic settings are easier, and items with words that are relatively more frequent in academic settings are more difficult. Difficult items have content that is rarely present in everyday settings and is instead more cultivated academically. Examining word frequency from different language settings can help researchers investigate test score interpretations and is a useful tool for predicting item difficulty and refining knowledge test items.