Measurement error in research on financial literacy: How much error is there and how does it influence effect size estimates?

Abstract

Financial literacy is often measured with only three to five questions, suggesting challenges with achieving respectable levels of internal consistency reliability. Based on our review, we found financial literacy tests composed of three and five test questions yielded mean reliability estimates of 0.40 (k = 7; N = 167,075) and 0.54 (k = 8; N = 57,937), respectively: values less than minimally acceptable for even exploratory research. Based on our more comprehensive review of 52 samples and a variety of financial literacy tests (3–45 questions), researchers are recommended to measure financial literacy with a minimum of 13–15 questions. Finally, we conclude that the potential impact of financial literacy on various outcome variables has been underestimated substantially in many previous investigations, as the relatively low levels of internal consistency reliability in the financial literacy test scores attenuated the obtained effects estimated from the observed scores.

Revalidating Fernandes et al.’s 2014 financial literacy scale in response to ongoing legislative and behavioral changes

“…Fernandes et al ( 2014)'s measure was created and validated via vigorous psychometric test-ing and achieves one of the highest internal consistency reliability scores of any financial literacy measure (Gignac and Ooi, 2022) and with that, it is one of the most important financial literacy measures both for academic research and for policy making. Nevertheless, we have shown how the questionnaire cannot continue to be used in its current format, as on-going changes in legislation invalidate one of its questions.…”

“…In an extensive review of different financial literacy measures, evaluating 71 individual studies, Huston (2010) suggested that the use of questionnaires with 12 to 20 items, measuring wider concepts with higher psychometric validity, is a better approach to the measurement of financial literacy. Similarly, Gignac and Ooi (2022) concluded that commonly used short questionnaires with only three or five questions have internal reliability values below what is minimally accepted and recommended a minimum of 13 to 15 questions.…”

“…One highly successful and more sophisticated financial literacy questionnaire is that of Fernandes et al (2014), which includes 13 questions of increasing levels of difficulty collated from other widely used scales, including questions from Lusardi and Mitchell. It was created and validated via vigorous psychometric testing and achieves one of the highest internal consistency reliability scores of any financial literacy measure (Gignac and Ooi, 2022), with Cronbach's α between 0.82 and 0.85 across three different studies. Its superiority is also further evidenced by its widespread use.…”

What do we really know about “don't know”? Re‐assessing the measurement of financial knowledge

“…Second, given that our financial knowledge scores were measured with a small number of measurement items (four to five), they may have suffered from low reliability. As discussed in Gignac and Ooi (2022), a low reliability in the scale could lead to underestimated coefficient estimates in regression analyses. Future research should assess how much reliability can be recovered with additional measurement items and which topics should be included to maintain a satisfactory degree of validity.…”

Explaining consumers' progress through life insurance decision states: The role of personal values and consumer characteristics

“…21 21 Limitations have been raised in terms of content coverage (Schmeiser & Seligman, 2013), focusing on knowledge rather than skills (Fernandes et al, 2014), and measurement error due to number of items (Gignac & Ooi, 2022). …”