A Methodology for Determining and Validating Latent Factor Dimensionality of Complex Multi-Factor Science Constructs Measuring Knowledge-In-Use

Kaldaras, Leonora; Akaeze, Hope; Krajcik, Joseph

doi:10.1080/10627197.2021.1971966

Cited by 12 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For our purpose, we recast response process validity as the extent to which the score levels assigned to children on the COR-Adv1.5 items reflect their level of proficiency on the construct being measured. Following Kaldaras et al (2021), we assessed this feature by examining the average factor scores against the observed (raw) score levels for each item. If the item score levels are well delineated and the scoring rubric is both properly defined in the test instrument and properly interpreted by the observers, the corresponding average factor score should be consistently higher as one progresses along the score levels for each item, and the separation between each raw score should be evident.…”

Section: Evidence Based On Response Processmentioning

confidence: 99%

Validation of the Child Observation Record Advantage 1.5 Assessment Tool for Preschool Children: A Multilevel Bifactor Modeling Approach

Akaeze

Lawrence

et al. 2023

Journal of Psychoeducational Assessment

View full text Add to dashboard Cite

This paper reports an investigation into the psychometric properties of the COR-Advantage1.5 (COR-Adv1.5) assessment tool, a criterion-referenced observation-based instrument designed to assess the developmental abilities of children from birth through kindergarten. Using data from 8534 children participating in a state-funded preschool program for disadvantaged 4-year-olds, we employed a multilevel bifactor model to show that CORAdv1.5 has some potential as an unbiased assessment tool for tracking developmental progress in pre-kindergarten children, under a bifactor model parameterization, with fewer dimensions than were proposed by the test developer, HighScope. Examination of factor adequacy indices revealed that, despite the multidimensional structure, CORAdv1.5 items overwhelmingly reflect variance due to the general factors at the child and classroom levels. Evidence for multilevel reliability, response process validity, predictive validity, longitudinal measurement invariance, and generalizability of the items are also reported. The findings of this study have important implications for the use of COR-Adv1.5.

show abstract

Section: Evidence Based On Response Processmentioning

confidence: 99%

Validation of the Child Observation Record Advantage 1.5 Assessment Tool for Preschool Children: A Multilevel Bifactor Modeling Approach

Akaeze

Lawrence

et al. 2023

Journal of Psychoeducational Assessment

View full text Add to dashboard Cite

show abstract

“…In terms of the type of teaching methods, hands-on projects are more meaningful and interesting for female students (Mitchell, 1993;Halpern, 2004;Geist and King, 2008). Project-based learning (Blumenfeld et al, 2000) has proven to increase students' engagement as well as a deeper understanding of scientific problems (Kaldaras et al, 2021). Instruction based on memorizing without understanding has become obsolete.…”

Section: Motivation: Curriculum Perception Perspectivementioning

confidence: 99%

I am done with this! Women dropping out of engineering majors

et al. 2022

View full text Add to dashboard Cite

Women are still underrepresented in STEM careers (Science, Technology, Engineering, and Mathematics). One of the possible drivers behind this gender gap in the labour market is the female dropout from STEM education. The causes of the gender differences in the persistence of pursuing STEM studies have been explained by multiple factors related to interest and resolution in this type of career. The goal of the present research is to study the Engineering persistence gender gap in higher education by exploring the main factors underlying the leakage in the pipeline of Engineering fields. Our study reports the results of 34 qualitative in-depth interviews where internal barriers, stereotypes and external obstacles are assessed by women who have left their university degrees, compared with men who have withdrawn and women who have persisted. Results from the content analysis suggest that the undermining of persistence in Engineering fields is related to factors such as the chilly and hostile environment in classes or the workload from an excessively demanding curriculum. Other factors affecting women’s withdrawal are the lack of role models and the perceived incongruity between the female gender role and STEM roles in society, leading to a weakening of female students’ self-efficacy and eroding their sense of belongingness, even making them consider dropping out of their Engineering degree. These findings provide information for the design of future STEM interventions aimed to enhance women’s persistence in STEM university studies.

show abstract

“…The student responses to the items were scored by a trained group of coders using holistic rubrics. The scores were then used to conduct measurement invariance (Kaldaras et al, 2021b) and item response theory analysis as part of the validation study which provided strong evidence for the validity of both LPs (Kaldaras, 2020;Kaldaras et al, 2021a).…”

Section: Study Settingmentioning

confidence: 99%

Rubric development for AI-enabled scoring of three-dimensional constructed-response assessment aligned to NGSS learning progression

Kaldaras

Yoshida²,

Haudek³

2022

Front. Educ.

Self Cite

View full text Add to dashboard Cite

IntroductionThe Framework for K-12 Science Education (the Framework) and the Next- Generation Science Standards (NGSS) define three dimensions of science: disciplinary core ideas, scientific and engineering practices, and crosscutting concepts and emphasize the integration of the three dimensions (3D) to reflect deep science understanding. The Framework also emphasizes the importance of using learning progressions (LPs) as roadmaps to guide assessment development. These assessments capable of measuring the integration of NGSS dimensions should probe the ability to explain phenomena and solve problems. This calls for the development of constructed response (CR) or open-ended assessments despite being expensive to score. Artificial intelligence (AI) technology such as machine learning (ML)-based approaches have been utilized to score and provide feedback on open-ended NGSS assessments aligned to LPs. ML approaches can use classifications resulting from holistic and analytic coding schemes for scoring short CR assessments. Analytic rubrics have been shown to be easier to evaluate for the validity of ML-based scores with respect to LP levels. However, a possible drawback of using analytic rubrics for NGSS-aligned CR assessments is the potential for oversimplification of integrated ideas. Here we describe how to deconstruct a 3D holistic rubric for CR assessments probing the levels of an NGSS-aligned LP for high school physical sciences.MethodsWe deconstruct this rubric into seven analytic categories to preserve the 3D nature of the rubric and its result scores and provide subsequent combinations of categories to LP levels.ResultsThe resulting analytic rubric had excellent human- human inter-rater reliability across seven categories (Cohen’s kappa range 0.82–0.97). We found overall scores of responses using the combination of analytic rubric very closely agreed with scores assigned using a holistic rubric (99% agreement), suggesting the 3D natures of the rubric and scores were maintained. We found differing levels of agreement between ML models using analytic rubric scores and human-assigned scores. ML models for categories with a low number of positive cases displayed the lowest level of agreement.DiscussionWe discuss these differences in bin performance and discuss the implications and further applications for this rubric deconstruction approach.

show abstract

A Methodology for Determining and Validating Latent Factor Dimensionality of Complex Multi-Factor Science Constructs Measuring Knowledge-In-Use

Cited by 12 publications

References 18 publications

Validation of the Child Observation Record Advantage 1.5 Assessment Tool for Preschool Children: A Multilevel Bifactor Modeling Approach

Validation of the Child Observation Record Advantage 1.5 Assessment Tool for Preschool Children: A Multilevel Bifactor Modeling Approach

I am done with this! Women dropping out of engineering majors

Rubric development for AI-enabled scoring of three-dimensional constructed-response assessment aligned to NGSS learning progression

Contact Info

Product

Resources

About