Implementation of Large‐Scale Education Assessments

Lietz, Petra; Cresswell, John C.; Rust, Keith; Adams, Raymond J.

doi:10.1002/9781118762462.ch1

Cited by 8 publications

(11 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…By the end of the 1990s, the number of participating countries was approximately 40 (Tijana & Anna, 2015). Nowadays, nearly 70% of countries across the world participate in these evaluations (Lietz, Cresswell, Rust & Adams, 2017). Table 1 shows a selection of recent ILSAs and the respective number of participating countries for reference.…”

Section: Introductionmentioning

confidence: 99%

Should different countries participating in PISA interpret socioeconomic background in the same way? A measurement invariance approach

Erylmaz¹,

Rivera-Gutiérrez

Sandoval-Hernández³

2020

RIEOEI

View full text Add to dashboard Cite

It has been claimed that there is a lack of theory-driven constructs and a lack of cross-country comparability in International Large-Scale Assessment (ILSA)’s socio-economic background scales. To address these issues, a new socio-economic background scale was created based on Pierre Bourdieu’s cultural reproduction theory, which distinguishes economic, cultural and social capital. Secondly, measurement invariance of this construct was tested across countries participating in the Programme for International Student Assessment (PISA). After dividing the countries which participated in PISA 2015 into three groups, i.e., Latin American, European, and Asian, a Multi-Group Confirmatory Factor Analysis was carried out in order to examine the measurement invariance of this new socio-economic scale. The results of this study revealed that this questionnaire, which measures the socio-economic background, was not found to be utterly invariant in the analysis involving all countries. However, when analysing more homogenous groups, measurement invariance was verified at the metric level, except for the group of Latin American countries. Further, implications for policymakers and recommendations for future studies are discussed.

show abstract

Section: Introductionmentioning

confidence: 99%

Should different countries participating in PISA interpret socioeconomic background in the same way? A measurement invariance approach

Erylmaz¹,

Rivera-Gutiérrez

Sandoval-Hernández³

2020

RIEOEI

View full text Add to dashboard Cite

show abstract

“…Lastly, a discussion of the results, and some limitations, of the study as well as recommendations for future assessments are provided. Lietz et al (2017) provide some insights to the key areas (or steps) that are required to be taken into account in the implementation of large-scale assessments to ensure the reliability and validity of the results obtained. Lietz et al (2017)'s 13 key areas are in also line with the four steps as suggested by Kirsch et al (2013) and these four steps will form the basis of the brief discussion of the development of the OECD/INFE financial literacy adult assessment in the remainder of this section.…”

Section: Sociodemographic Informaɵonmentioning

confidence: 99%

“…To ensure the internationally comparability of an assessment, it is essential to have agreement on the concept to be measured. Agreement should also be achieved in the operationalisation of the concept through the development and application of a measurement instrument that provides fully cross-country comparable results (Kirsch et al 2013;Lietz et al 2017). Informed by the definition of financial literacy, the suggested assessment framework is illustrated in Figs.…”

Section: Step 1: Policy Questionsmentioning

confidence: 99%

“…As stated by Kirsch et al (2013), ILSA studies have "expanded in scope over time in response to increasing concern about the distribution of human capital and the growing recognition that skills contribute to the prosperity of nations and to better lives for individuals in those nations". Given the ethics, and dependence, of a variety of stakeholders on the information collected in these ILSAs, Lietz et al (2017) suggest that these large-scale assessments should be robust and useful, of high quality, technically sound, have a comprehensive communication strategy and be useful for education policy.…”

mentioning

confidence: 99%

See 1 more Smart Citation

A comparative analysis of the OECD/INFE financial knowledge assessment using the Rasch model

Clercq

2019

Empirical Res Voc Ed Train

View full text Add to dashboard Cite

“…Another main focus of attention lies on changes in achievement over time. Trend estimations serve at least two purposes: First, they help estimate the impact of interventions and, second, they reveal how patterns of achievement between countries or subgroups develop over time (Lietz, Cresswell, Rust, & Adams, 2017).…”

mentioning

confidence: 99%

When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

Sachse

Mahler

Pohl

2019

Educational and Psychological Measurement

View full text Add to dashboard Cite

Mechanisms causing item nonresponses in large-scale assessments are often said to be nonignorable. Parameter estimates can be biased if nonignorable missing data mechanisms are not adequately modeled. In trend analyses, it is plausible for the missing data mechanism and the percentage of missing values to change over time. In this article, we investigated (a) the extent to which the missing data mechanism and the percentage of missing values changed over time in real large-scale assessment data, (b) how different approaches for dealing with missing data performed under such conditions, and (c) the practical implications for trend estimates. These issues are highly relevant because the conclusions hold for all kinds of group mean differences in large-scale assessments. In a reanalysis of PISA (Programme for International Student Assessment) data from 35 OECD countries, we found that missing data mechanisms and numbers of missing values varied considerably across time points, countries, and domains. In a simulation study, we generated data in which we allowed the missing data mechanism and the amount of missing data to change over time. We showed that the trend estimates were biased if differences in the missing-data mechanisms were not taken into account, in our case, when omissions were scored as wrong, when omissions were ignored, or when model-based approaches assuming a constant missing data mechanism over time were used. The results suggest that the most accurate estimates can be obtained from the application of multiple group models for nonignorable missing values when the amounts of missing data and the missing data mechanisms changed over time. In an empirical example, we furthermore showed that the large decline in PISA reading literacy in Ireland in 2009 was reduced when we estimated trends using missing data treatments that accounted for changes in missing data mechanisms.

show abstract

Implementation of Large‐Scale Education Assessments

Cited by 8 publications

References 14 publications

Should different countries participating in PISA interpret socioeconomic background in the same way? A measurement invariance approach

Should different countries participating in PISA interpret socioeconomic background in the same way? A measurement invariance approach

A comparative analysis of the OECD/INFE financial knowledge assessment using the Rasch model

When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments

Contact Info

Product

Resources

About