Invariance of the Response Processes Between Gender and Modes in an Assessment of Reading

Kroehne, Ulf; Hahnel, Carolin; Goldhammer, Frank

doi:10.3389/fams.2019.00002

Cited by 11 publications

(4 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, as more than just the (overall) number of missing values seems relevant, further research is necessary to fully understand the empirical and theoretical role of missing value coding in the context of comparisons across modes and over time in terms of trends. As we did not compare test speed between modes, we see the need to investigate time‐related differences between modes in the future (see, e.g., Kroehne, Hahnel & Goldhammer, ).…”

Section: Discussionmentioning

confidence: 99%

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Kroehne

Buerger

Hahnel

et al. 2019

Educational Measurement

Self Cite

View full text Add to dashboard Cite

For many years, reading comprehension in the Programme for International Student Assessment (PISA) was measured via paper‐based assessment (PBA). In the 2015 cycle, computer‐based assessment (CBA) was introduced, raising the question of whether central equivalence criteria required for a valid interpretation of the results are fulfilled. As an extension of the PISA 2012 main study in Germany, a random subsample of two intact PISA reading clusters, either computerized or paper‐based, was assessed using a random group design with an additional within‐subject variation. The results are in line with the hypothesis of construct equivalence. That is, the latent cross‐mode correlation of PISA reading comprehension was not significantly different from the expected correlation between the two clusters. Significant mode effects on item difficulties were observed for a small number of items only. Interindividual differences found in mode effects were negatively correlated with reading comprehension, but were not predicted by basic computer skills or gender. Further differences between modes were found with respect to the number of missing values.

show abstract

Section: Discussionmentioning

confidence: 99%

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Kroehne

Buerger

Hahnel

et al. 2019

Educational Measurement

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using response times from ILSAs in a joint modeling framework with response accuracy, Kroehne et al (2019) evaluated the invariance of response processes regarding the assessment mode (i.e., computer-based vs. paper-based) and the respondent's gender. With an online publication appearing in 2019, the work of Ulitzsch et al (2020a), in turn, aimed to develop a framework that incorporates respondent's nonresponse behavior to gain a deeper understanding of the processes underlying item omissions in large-scale assessments.…”

Section: Group Of Items/test-level Analysis and Personal Characterist...mentioning

confidence: 99%

Process Data Analysis in ILSAs

Costa

Netto²

2022

International Handbook of Comparative Large-Scale Studies in Education

View full text Add to dashboard Cite

Computational advancements in the last couple of decades have brought forth a new era of international large-scale assessments (ILSAs) in which the administration of computer-based tests is becoming the norm. Beyond collecting the correct/incorrect answers for each item, computerbased assessments are also able to collect a range of actions performed by the respondents in the computer testing application during the course of the test administration. Both respondents' actions-starting a unit, clicking a button, spending time until inputting or submitting an answer, and so forth-and their overall behavior-what they do with their keyboards, mice, and even their own eyes-are recorded as a new set of data called "process data", which are normally timestamped and can be stored in so-called log files. There may be plenty of useful insight into the respondent's cognitive process, and process data can potentially become a relevant element in the scoring process of an assessment, validate test score interpretations, to name a few possibilities for the analysis of such data. This chapter aims to contribute to the body of knowledge in this area by offering (1) an introduction to process data, what kind of data it contains, its relation to the cognitive process and how it can be organized into a proposed 6-layered ecological framework that facilitates its analysis; (2) a literature review of 37 seminal and state-of-the-art publications produced in the last decade on the topic, which are then analyzed both in their chronological perspective as well as in how they fit into the ecological framework; and (3) a discussion of the potential and limitations of using process data in the assessment framework, including a view of what could be the next steps in the analysis of process data from ILSAs.

show abstract

“…As the use and importance of computers increases in society and education, it seems more relevant to assess competencies in CBA than in PBA mode, and it could therefore be argued that it is not necessarily of interest how a trend would continue if it were based on a PBA. Accordingly, future research should investigate potential sources for mode effects, such as mode-related speed differences (Kroehne et al, 2019b).…”

Section: Should Mode Effects Be Adjusted At All?mentioning

confidence: 99%

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

et al. 2020

Self Cite

View full text Add to dashboard Cite

International large-scale assessments, such as the Program for International Student Assessment (PISA), are conducted to provide information on the effectiveness of education systems. In PISA, the target population of 15-year-old students is assessed every 3 years. Trends show whether competencies have changed in the countries between PISA cycles. In order to provide valid trend estimates, it is desirable to retain the same test conditions and statistical methods in all PISA cycles. In PISA 2015, however, the test mode changed from paper-based to computer-based tests, and the scaling method was changed. In this paper, we investigate the effects of these changes on trend estimation in PISA using German data from all PISA cycles (2000)(2001)(2002)(2003)(2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011)(2012)(2013)(2014)(2015). Our findings suggest that the change from paper-based to computer-based tests could have a severe impact on trend estimation but that the change of the scaling model did not substantially change the trend estimates.

show abstract

Invariance of the Response Processes Between Gender and Modes in an Assessment of Reading

Cited by 11 publications

References 32 publications

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

Process Data Analysis in ILSAs

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects

Contact Info

Product

Resources

About