Students’ Complex Problem Solving Profiles

Abstract

Complex problem solving (CPS) is an up-and-coming twenty-first century skill that requires testtakers to solve dynamically changing problems, often assessed using computer-based tests. The log data that users produce when interacting with a computer-based test provide valuable information about each individual behavioral action they undertake, but such data are rather difficult to handle from a statistical point of view. This paper addresses this issue by building upon recent research focused on decoding log data and aims to identify homogeneous student profiles with regard to their ability to solve CPS tasks. Therefore, we estimated a discrete two-tier item response theory model, which allowed us to profile units (i.e., students) while taking into account the multidimensionality of the data and the explanatory effect of individual characteristics. The results indicate that: (1) CPS can be thought of as a three-dimensional latent variable; (2) there are ten latent classes of students with homogenous profiles regarding the CPS dimensions; (3) students in the higher latent classes generally demonstrate higher cognitive and noncognitive performances; (4) some of the latent classes seem to profit from learning-by-doing within tasks, whereas others seem to exhibit the reverse behavior; (5) cognitive and non-cognitive skills, as well as gender and to some extent age, contribute to distinguishing among the latent classes.

Analysing Complex Problem-Solving Strategies from a Cognitive Perspective: The Role of Thinking Skills

“…LCA is a latent variable modelling approach that can be used to identify unmeasured (latent) classes of samples with similarly observed variables. LCA has been widely used in analysing logfile data for CPS assessment and in exploring students’ behaviour patterns (see, e.g., Gnaldi et al 2020 ; Greiff et al 2018 ; Molnár et al 2022 ; Molnár and Csapó 2018 ; Mustafić et al 2019 ; Wu and Molnár 2021 ). The scores for the use of VOTAT in the KAC phase (0, 1, 2; see Section 3.4 ) were used for the LCA analysis.…”

Assessing Complex Problem-Solving Skills in Under 20 Minutes

“…Tasks based on the MicroDYN approach attempt to measure the two processes independently in two phases by providing the full solution to the knowledge acquisition phase (i.e., the relation between input and output variables) in the knowledge application phase. Empirically, however, very high correlations (>.85) are reported between knowledge acquisition and knowledge application (e.g., Gnaldi et al, 2020;Wüstenberg et al, 2012). Other publications even regressed the two factors onto a single higher-order factor of CPS (e.g., Greiff & Fischer, 2013;Kretzschmar et al, 2014) with knowledge acquisition consistently showing the stronger loadings.…”

Skills required to solve complex problems and their assessment and development in secondary school

“…Seda võib vaadelda komplekssete probleemide lahendamise oskusena, sest viimane eeldab kompleksse probleemi lahtiharutamist osaprobleemideks ning see järel sobiva spetsiifilisema probleemilahendamisoskuse rakendamist. Vara semad uuringud on komplekssete probleemide lahendamise oskust käsitlenud ainespetsiifiliste keeru kate probleemide lahendamise oskusena (vt Herde et al, 2016;Sternberg & Frensch, 2014) või aineteülese konstruktina, mis koosneb üldistest probleemi lahendamisoskustest (vt Gnaldi et al, 2020), aga siinne uuring avab selle erinevate probleemilahendamisstrateegiate ühisosa leidmise vaatepunktist. Nii näiteks võib ühe kompleksse probleemi lahendamisel olla vaja nii matemaatilist, algoritmilist kui ka uurimuslikku lähenemist.…”

“…Selles on välja toodud, et siiani keskenduvad empiirilised uuringud kompleks sete probleemide lahendamisel ühe spetsiifilise strateegia uurimisele ning eri strateegiate rakendamise ühisosa üldoskusena jääb tuleviku uuringute lahendatavaks probleemiks. Sama uurimisrühma päris värske uuring (Gnaldi, Bacci, Kunze, & Greiff, 2020) keskendub komplekssete prob leemide lahendamise oskuse hindamisel logifailide analüüsile ja käsitleb seda kui latentset mitme dimensioonilist tunnust. Eristatakse probleemi uurimist, teadmiste omandamist ja teadmiste rakendamist kui üldisi probleemi lahendamise protsesse, kuid probleemi lahendamise strateegia on siiski üks (kuigi vald kondadeülene).…”