Convergent Evidence for the Validity of a Performance-Based ICT Skills Test

Abstract

Abstract. The goal of this study was to investigate sources of evidence of convergent validity supporting the construct interpretation of scores on a simulation-based information and communication technology (ICT) skills test. The construct definition understands ICT skills as reliant on ICT-specific knowledge as well as comprehension and problem-solving skills. On the basis of this, a validity argument comprising three claims was formulated and tested. (1) In line with the classical nomothetic span approach, all three predictor variables explained task success positively across all ICT skills items. As ICT tasks can vary in the extent to which they require construct-related knowledge and skills and in the way related items are designed and implemented, the effects of construct-related predictor variables were expected to vary across items. (2) A task-based analysis approach revealed that the item-level effects of the three predictor variables were in line with the targeted construct interpretation for most items. (3) Finally, item characteristics could significantly explain the random effect of problem-solving skills, but not comprehension skills. Taken together, the obtained results generally support the validity of the construct interpretation.

Development and Evaluation of a Framework for the Performance-Based Testing of ICT Skills

“…First, the fact that the definition of ICT skills refers to other constructs suggests analyzing convergent sources of validity evidence, namely the relations with ICT-specific variables as well as problem-solving and reading comprehension (validity evidence based on relation to other variables; American Educational Research Association, American Psychological Association, and National Council on Measurement in Education [AERA, APA and NCME], 2014). In an empirical study (Engelhardt et al, 2020), positive relations between ICT skills items and reading and problem-solving skills as well as technical knowledge were reported. As expected, the effect of technical knowledge tended to be stronger for harder items, but not the effect of the conventional skills of reading and problem-solving skills, supporting the assumption that ICT-specific knowledge rather than reading or problem-solving skills requirements should drive item difficulty.…”

Qualitative Analysis of Use of ICTs and Necessary Personal Competencies (Self-Efficacy, Creativity and Emotional Intelligence) of Future Teachers: Implications for Education

“…Teachers' improvement in ICT competencies is an indispensable requirement in education, so the need for integral training is obvious [14]. Moreover, ICT literacy has become a basic competence for teachers who must be able to prepare their students to master new technologies [15]. The Technological Pedagogical Content Knowledge model [16] emphasises the importance of addressing technology knowledge, content, and pedagogy together.…”