How Well Does the Sum Score Summarize the Test? Summability as a Measure of Internal Consistency

Abstract

Many researchers use Cronbach's alpha to demonstrate internal consistency, even though it has been shown numerous times that Cronbach's alpha is not suitable for this. Because the intention of questionnaire and test constructers is to summarize the test by its overall sum score, we advocate summability, which we define as the proportion of total test variation that is explained by the sum score. This measure is closely related to Loevinger's H. The mathematical derivation of summability as a measure of explained variation is given for both scale and dichotomously scored items. Using computer simulations, we show that summability performs adequately and we apply it to an existing productive vocabulary test. An open‐source tool to easily calculate summability is provided online (https://sites.google.com/view/summability).

Moving the field of vocabulary assessment forward: The need for more rigorous test development and validation

“…Reliability has typically been measured by ‘internal consistency’ statistics, like Cronbach's alpha, but we feel there are two reasons for not doing this. One, Cronbach's alpha has been criticized for mainly being an indicator of how many items a test contains (Goeman & de Jong, 2018). Vocabulary tests usually have many items, which renders this index less informative.…”

“…some combination of incidental exposure and instruction), it is best to establish reliability using a test-retest procedure (see Schmitt, 2010 for more details). Alternatively, newer approaches, such as the concept of summability (the ‘proportion of total test variation that is explained by the sum score’ (Goeman & de Jong, 2018, p. 54) might be incorporated into vocabulary validation procedures.…”

Factors Influencing the Maternal Life Balance of Flemish Mothers, a Cross-Sectional Study

“…To examine the proportion of total test variation explained by the summed score, we calculated the summability of this category using online software (https://goeman.shinyapps.io/summability/). A summability of 1 indicated perfectly correlated items and summability around zero indicated inadequate item-correlation (Goeman & de Jong, 2018). Kruskal-Wallis tests was used to examine the differences in mean life balance ranks of mothers from various age groups and of mothers with children in different age groups.…”

“…Cronbach's alpha showed overall an acceptable internal relatedness of the (sub)scales' items (Table 3). The maternal life balance category showed an acceptable summability of .72 (Goeman & de Jong, 2018). The BNPS, SOC-13 and COPE-Easy showed test score variances as the BPNS competence subscale, the SOC-13 manageability and meaning subscales and the COPE-Easy avoidance and positive thinking subscales showed a <.70 (Skorupski & Carvajal, 2010).…”

Elicited Imitation: A Test for All Learners?

“…The EI test showed good model fit (Mean Infit MnSq = 1.04, SD = .31), and differentiated significantly (χ 2 (38) = 3,900.4, p < .000) between learners across a wide proficiency spectrum (Separation = 11.43; Strata = 15.57; Reliability = .99). The internal consistency and reliability of the EI test was further analyzed by calculating summability as a measure of internal consistency (Goeman & De Jong, 2018) and Cronbach's alpha as a measure of reliability-both with satisfactory outcomes (summability = .46, α = .966).…”