Methodological challenges in international comparative post-secondary assessment programs: lessons learned and the road ahead

Wolf, Raffaela; Zahner, Doris; Benjamin, Roger

doi:10.1080/03075079.2015.1004239

Cited by 21 publications

(20 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The test has an open-ended response format and is complemented by 25 selected-response questions on separate item stems. According to Wolf et al (2015), the assessment measures the following student abilities: Problem-solving and analysis, writing effectiveness, writing mechanics, reasoning scientifically and quantitatively, reading critically and evaluatively, and critiquing an argument.…”

Section: Assessment Framework For Measuring Critical Reasoningmentioning

confidence: 99%

The Role of Students’ Beliefs When Critically Reasoning From Multiple Contradictory Sources of Information in Performance Assessments

Zlatkin‐Troitschanskaia

Beck

Fischer

et al. 2020

Front. Psychol.

View full text Add to dashboard Cite

Critical reasoning (CR) when confronted with contradictory information from multiple sources is a crucial ability in a knowledge-based society and digital world. Using information without critically reflecting on the content and its quality may lead to the acceptance of information based on unwarranted claims. Previous personal beliefs are assumed to play a decisive role when it comes to critically differentiating between assertions and claims and warranted knowledge and facts. The role of generic epistemic beliefs on critical stance and attitude in reflectively dealing with information is well researched. Relatively few studies however, have been conducted on the influence of domain-specific beliefs, i.e., beliefs in relation to specific content encountered in a piece of information or task, on the reasoning process, and on how these beliefs may affect decision-making processes. This study focuses on students' task-and topicrelated beliefs that may influence their reasoning when dealing with multiple and partly contradictory sources of information. To validly assess CR among university students, we used a newly developed computer-based performance assessment in which the students were confronted with an authentic task which contains theoretically defined psychological stimuli for measuring CR. To investigate the particular role of task-and topic-related beliefs on CR, a purposeful sample of 30 university students took part in a performance assessment and then were interviewed immediately afterward. In the semistructured cognitive interviews, the participants' task-related beliefs were assessed. Based on qualitative analyses of the interview transcripts, three distinct profiles of decision-making among students have been identified. More specifically, the different types of students' beliefs and attitudes derived from the cognitive interview data suggest their influence on information processing, reasoning approaches and decision-making. The results indicated that the students' beliefs had an influence on their selection, critical evaluation and use of information as well as on their reasoning processes and final decisions.

show abstract

Section: Assessment Framework For Measuring Critical Reasoningmentioning

confidence: 99%

The Role of Students’ Beliefs When Critically Reasoning From Multiple Contradictory Sources of Information in Performance Assessments

Zlatkin‐Troitschanskaia

Beck

Fischer

et al. 2020

Front. Psychol.

View full text Add to dashboard Cite

show abstract

“…The CLA has been very influential in setting the standards for the test‐based measurement of learning outcomes in the US. Developed by the Council for Aid to Education (CAE), the CLA is a test with open‐ended questions to measure higher‐order skills, such as analytic reasoning and evaluation, problem‐solving and written communication (Wolf, Zahner, & Benjamin, ). An alternative test‐based approach based on multiple‐choice questions in combination with open‐ended questions has been developed by the Australian Council for Educational Research (ACER).…”

Section: How: Grades Self‐reported and Test‐based Measuresmentioning

confidence: 99%

Measuring learning outcomes

Caspersen

Smeby

Aamodt

2017

Euro J of Education

View full text Add to dashboard Cite

The growing interest for measurement of learning outcomes relates to long lines of development in higher education, the request for accountability, intensified through international reforms and movements such as the development and implementation of qualifications frameworks. In this article, we discuss relevant literature on different approaches to measurement and how learning outcomes are measured, what kinds of learning outcomes are measured, and why learning outcomes are measured. Three dimensions are used to structure the literature: Whether the approaches emphasise generic or disciplinary skills and competence, self-assessment or more objective test based measures (including grades), and how the issue of the contribution from the education program or institution (the value-added) are discussed. It is pointed out that large scales initiatives that compare institutions and even nations seem to fall short because of the implicit and explicit differences in context, whilst small-scale approaches suffer from a lack of relevance outside local contexts. In addition, competence (actual level of performance) is often confused with learning (gain and development) in many approaches, laying the ground for false assumptions about institutional process-quality in higher education.K E Y W O R D S assessment, higher education, knowledge, learning, learning outcomes, measurement

show abstract

“…In these terms, students' gains will be a more representative indicator for the quality of education in the university. Another point of criticism was the question about cross-cultural validity of AHELO instruments (Pearce, 2014;Wolf et al 2015).…”

Section: Introductionmentioning

confidence: 99%

Constructing Tests that Can Measure and Compare the Maths and Physics Skills of Engineering Students in Russia and China

et al. 2015

View full text Add to dashboard Cite

Although the number of engineering graduates has expanded rapidly in the last two decades, relatively little is known about the quality of engineering programs worldwide. In particular, few studies look at differences in the degree to which students are learning skills across different engineering programs within and between countries. There is particular interest in the investigation of the engineering education quality in the countries with the rapidly growing economy, such as BRICS countries. Until now, there was little research in this field and one of the main reasons for this is the difficulty in developing an assessment approach and the accompanying set of instruments, which would allow for measurement and international comparison. Our study describes a set of procedures for developing such an assessment framework of instruments, to measure and compare skill levels and gains across engineering programs. We first describe a systematic approach for constructing cross-nationally comparable instruments in maths and physics for students in the first two years of their undergraduate engineering programs. The approach includes both a priori procedures (including expert assessments to avoid construct, method, and item bias), and a posteriori procedures (including the psychometric analysis of test quality, differential item functioning, and identifying and reducing bias in the data). In addition to describing this set of procedures in theory, we also show how we systematically implemented these procedures. Drawing on data that we collected from over 24 engineering experts and 3,600 engineering students across Russia and China, we provide evidence that it is possible to create tests that are cross-culturally valid, equate-able, and free from bias.

show abstract

Methodological challenges in international comparative post-secondary assessment programs: lessons learned and the road ahead

Cited by 21 publications

References 19 publications

The Role of Students’ Beliefs When Critically Reasoning From Multiple Contradictory Sources of Information in Performance Assessments

The Role of Students’ Beliefs When Critically Reasoning From Multiple Contradictory Sources of Information in Performance Assessments

Measuring learning outcomes

Constructing Tests that Can Measure and Compare the Maths and Physics Skills of Engineering Students in Russia and China

Contact Info

Product

Resources

About