Validation of an Instrument for Measuring Students’ Understanding of Interdisciplinary Science in Grades 4-8 over Multiple Semesters: a Rasch Measurement Study

Yang, Yang; He, Peng; Liu, Xiufeng

doi:10.1007/s10763-017-9805-7

Cited by 26 publications

(22 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also generated the WLE person separation index, which estimates if the order of person abilities could be replicated with a different set of items of similar difficulty (Bond & Fox, ). Values > 0.70 are considered acceptable (Grigg & Manderson, ; Yang, He, & Liu, ).…”

Section: Methodsmentioning

confidence: 99%

Do you see what I‐SEA? A Rasch analysis of the psychometric properties of the Inventory of Student Evolution Acceptance

Sbeglia

Nehm

2019

Science Education

View full text Add to dashboard Cite

Theoretical perspectives and empirical research suggest that acceptance of evolution may be contingent upon what is evolving (e.g., plants vs. humans) and the scale of change (microevolution vs. macroevolution). The Inventory of Student Evolution Acceptance (I-SEA) is the only instrument designed to measure acceptance at different evolutionary scales and in different evolutionary contexts. Nevertheless, current validity testing for this instrument remains limited and grounded in Classical Test Theory. In this study, we examine patterns of evolution acceptance using the I-SEA instrument in a large sample of undergraduates (n > 2,000 participants) from six semesters of an evolution-focused biology class. We examine three research questions: (RQ1) Does Rasch analysis support the psychometric properties of the I-SEA discussed by Nadelson and Southerland? (RQ2) How variable are I-SEA measures across semesters, and are they sensitive to evolution instruction? (RQ3) Are I-SEA measures comparable to evolution acceptance measures produced by the Generalized Acceptance of EvolutioN Evaluation? Our empirical results support theoretical claims that evolution acceptance does not appear to be a unidimensional construct among novice learners. We identify limitations of the I-SEA and recommend multiple modifications to improve instrument quality.

show abstract

Section: Methodsmentioning

confidence: 99%

Do you see what I‐SEA? A Rasch analysis of the psychometric properties of the Inventory of Student Evolution Acceptance

Sbeglia

Nehm

2019

Science Education

View full text Add to dashboard Cite

show abstract

“…We also generated the WLE person separation index, which estimates if the order of person abilities could be replicated with a different set of items of similar difficulty (Bond and Fox 2001). Reliability values of greater than 0.70 are considered acceptable (Grigg and Manderson 2016;Yang et al 2017).…”

Section: Methodsmentioning

confidence: 99%

Measuring evolution acceptance using the GAENE: influences of gender, race, degree-plan, and instruction

Sbeglia

Nehm

2018

Evo Edu Outreach

View full text Add to dashboard Cite

Background: The evolution education research community has defined the construct of "evolution acceptance" in different ways and measured it using different instruments. One of these instruments-the GAENE-has not been analyzed across different student populations, demographic groups, degree plans, and instructional treatments. Such comparisons are crucial for examining whether the inferences drawn from instrument measures are valid, reliable, and generalizable. In this study, we attempt to replicate findings produced in the original validation study and explore aspects of the instrument not previously examined. Methods: We use Rasch analysis to study a large sample (n > 700) of undergraduates enrolled in standard introductory biology classes in the Northeastern USA. Participants completed the GAENE pre-and post-course for two semesters, and the MATE pre-and post-course for one semester. We assessed dimensionality, reliability, item fit, and rating scale functioning. We used regression analyses and generalized eta squared to evaluate the contribution of demographic and background variables to pre-course measures and pre-post course acceptance gains. Results: Our analyses of GAENE dimensionality and item properties were generally in line with prior work, including the finding that particular items displayed psychometric problems. Surprisingly, GAENE measures did not differ between biology majors and non-majors. Evolution instruction produced significant but small pre-post improvements in GAENE measures. GAENE measures were significantly associated with MATE measures (0.68-0.80). White and male participants had the highest evolution acceptance measures using both the MATE and the GAENE; race had a much stronger contribution to MATE measures as compared to GAENE measures. Race and gender acceptance differences were found to be as large as the differences produced in response to evolution instruction. Conclusions: Overall measures of acceptance change will be similar, but not identical, using the MATE and the GAENE. We make several recommendations for the modification or removal of some GAENE items, as well as future research directions for the measurement of evolution acceptance.

show abstract

“…Rasch model is used as a better way to convert raw scores into ratio scores so that people's abilities can be measured on a ratio scale. Several studies have used Rasch modeling to validate the instruments used (He et al, 2016;Yang et al, 2018). The validity of the SRAVZ instrument can be seen from the reliability and separation index, unidimensionality, item polarity, fit and misfit items, and items' difficulty level.…”

Section: Methodsmentioning

confidence: 99%

“…The instrument reliability can be seen from the items' completeness based on the students' abilities and the clarity of the item test's meaning. Validation is the collection of evidence to provide a scientific basis of test scores interpretation (Yang et al, 2018). Also, validation is the process of generating and interpreting evidence to conclude that the tool used matches the attributes that will be measured (Peeters & Martin, 2017).…”

Section: Introductionmentioning

confidence: 99%

Self-Reflection Assessment in Vertebrate Zoology (SRAVZ) Using Rasch Analysis

Yuhanna

Muhdhar

Gofur

et al. 2021

JPII

View full text Add to dashboard Cite

Instruments that are valid, reliable, and have high consistency are needed to measure studentsâ€™ self-reflection. The Self-Reflection Assessment in Vertebrate Zoology (SRAVZ) was developed to explore studentsâ€™ self-reflection and abilities in the vertebrate zoology course. It is essential to test the instrumentâ€™s validity before measuring studentsâ€™ abilities so that data bias does not occur. This study aims to determine the validity, whether the items are fit or misfit, and the difficulty level of SRAVZ items. SRAVZ is developed by ADDIE (Analysis, Design, Development, Implementation, Evaluation). The SRAVZ consists of 24 items tested on 135 students who have taken the vertebrate zoology course. Analysis of the Rasch model using Winstep version 4.5.2. The Rasch model shows the item reliability value at 0.97. The Cronbach alpha value at 0.94 with PTMEA Corr shows a positive value, unidimensional 48.1%. The separation index of 5.6 means that the level of grouping the items is very good. The mean square infit for SRAVZ was 0.59-1.96, and the mean square outfit value is 0.59-2.16. Data analysis shows that 24 SRAVZ items have 22 fit items and two misfit items (S3 and S5). Item numbers S3 and S5 must be excluded from the SRAVZ construction. Total items used to measure studentsâ€™ self-reflection in the vertebrate zoology course were 22 items. The most difficult item is S3, and the easiest item is S6. Thus, the data indicate that the valid and reliable SRAVZ is in the good, effective, and high level of consistency category

show abstract

Validation of an Instrument for Measuring Students’ Understanding of Interdisciplinary Science in Grades 4-8 over Multiple Semesters: a Rasch Measurement Study

Cited by 26 publications

References 21 publications

Do you see what I‐SEA? A Rasch analysis of the psychometric properties of the Inventory of Student Evolution Acceptance

Do you see what I‐SEA? A Rasch analysis of the psychometric properties of the Inventory of Student Evolution Acceptance

Measuring evolution acceptance using the GAENE: influences of gender, race, degree-plan, and instruction

Self-Reflection Assessment in Vertebrate Zoology (SRAVZ) Using Rasch Analysis

Contact Info

Product

Resources

About