Sources of difficulty in assessment: example of PISA science items

Hebel, Florence Le; Montpied, Pascale; Tiberghien, Andrée; Fontanieu, Valérie

doi:10.1080/09500693.2017.1294784

Cited by 24 publications

(23 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Items on the PISA 2015 science assessment were developed using the PISA 2015 science framework which delineates four aspects of scientific literacy, namely contexts, knowledge, competencies, and attitudes (OECD, 2017a). However, some critics of PISA have raised concerns regarding the content and substance validity of the PISA science assessment items (Lau, 2009; Le Hebel, Montpied, Tiberghien, & Fontanieu, 2017; Nentwig, Roennebeck, Schoeps, Rumann, & Carstensen, 2009). How well the PISA science assessment items really address scientific literacy, rather than content knowledge, is a controversial issue.…”

Section: Methodsmentioning

confidence: 99%

“…Although the stated construct definitions of PISA scientific literacy measures are good, there have been concerns raised about the extent to which the questions in any one cycle match the proposed definition (Lau, 2009), and the extent to which the questions properly elicit students' reasoning with respect to scientific literacy (Le Hebel, Montpied, & Tiberghien, 2014; Nentwig et al, 2009). Hence, the current study used the term “science achievement” rather than “scientific literacy.” Yet another criticism of the PISA science assessment items questions the difficulty of these items for many students due to cognitive demand required to respond to the item (Le Hebel et al, 2017) and possible clues or inferences drawn from the item contexts (Nentwig et al, 2009). The difficulty of these items becomes exaggerated for students of relatively lower prior achievement.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Students' attitudes toward science and science achievement: An analysis of the differential effects of science instructional practices

Liou

2020

J Res Sci Teach

View full text Add to dashboard Cite

The purpose of this study was to examine the predictive effects of inquiry‐based and teacher‐directed instructional practices on students' attitudes toward science and science achievement in a nationally representative sample of 7,708 15‐year‐old‐students from Taiwan who took part in the sixth cycle of the Program for International Student Assessment (PISA). The results of structural equation modeling analyses, after accounting for student demographic and socioeconomic characteristics, revealed that teacher‐directed instructional practices had a significant positive effect on students' science achievement, whereas inquiry‐based instructional practices had a significant negative effect on students' science achievement. Moreover, the results of the study indicated that inquiry‐based instructional practices had greater positive predictive power than teacher‐directed instructional practices for students' attitudes toward science. The inverse relations between inquiry‐based science instructional practices and student science achievement contradicts the popular belief that inquiry‐based science instructional practices help improve student science achievement. However, the results of the current study underscore the positive effects of inquiry‐based instructional practices on students' attitudes toward science. Indeed, optimizing the frequency of instructional practices in science classrooms to promote the desired educational outcomes deserves further discussion.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Students' attitudes toward science and science achievement: An analysis of the differential effects of science instructional practices

Liou

2020

J Res Sci Teach

View full text Add to dashboard Cite

show abstract

“…There is evidence that, in attempting to make sense of items, examinees make connections between the contexts of items and their own personal experiences Li, 2009, 2013). Item contexts are more meaningful to examinees when they portray situations in which they are actors, rather than observers or apprentices (Solano-Flores and Nelson-Barber, 2001;Le Hebel et al, 2013). An implication of this evidence is that, if the situations and lifestyles portrayed by items are predominantly those of a given cultural group, then contexts may fail to provide the same level of support to all students, even if those items are seemingly familiar to all.…”

Section: Item Semiotic Alignmentmentioning

confidence: 99%

The Semiotics of Test Design: Conceptual Framework on Optimal Item Features in Educational Assessment Across Cultural Groups, Countries, and Languages

Solano‐Flores

2021

Front. Educ.

View full text Add to dashboard Cite

This paper offers a conceptual framework on test design from the perspective of social semiotics. Items are defined as arrangements of features intended to represent information, convey meaning, and capture information on the examinees’ knowledge or skills on a given content. The conceptual framework offers a typology of semiotic resources used to create items and discusses item representational complexity—the multiple ways in which the semiotic resources of an item are related to each other—and item semiotic alignment—the extent to which examinees share cultural experience encoded by items. Since the ability to make sense of items is shaped by the examinees’ level of familiarity with the social conventions underlying the ways in which information is represented, unnecessary representational complexity and limited semiotic alignment may increase extraneous item cognitive load and adversely impact the performance of examinees from certain populations. Semiotic test design allows specification of optimal pools of semiotic resources to be used in creating items with the intent to minimize representational complexity and maximize semiotic alignment for the maximum number of individuals in diverse populations of examinees. These pools of semiotic resources need to be specific to the content assessed, the characteristics of the populations of examinees, the languages involved, etc., and determined based on information produced by cross-cultural frequency analyses, cognitive interviews, focus groups, and expert panels.

show abstract

“…The first is the "basic" category which relate to contents that students have learnt in their earlier years and are therefore more familiar with. Such items may not be difficult for many students because of repeated engagement with them (Berger, Bowie and Nyaumwe 2010;Le Hebel et al 2017). Next are level-appropriate contents which the author termed "adequate", followed by those termed "advanced" because they relate to knowledge elements that will be given greater depth subsequently (Leong 2006).…”

Section: Levels Of Difficulty (Lod)mentioning

confidence: 99%

“…However, the outcome based on familiarity provided a wider spread and hence cognitive variation in this section will be discussed based on familiarity. Studies have shown that familiarity imparts the CD of assessment tasks (Le Hebel et al 2017). Where students have had several opportunities to work with a particular concept, procedure or fact, they become conversant with such knowledge elements and are unlikely to find them challenging.…”

Section: Understand-content-hybridmentioning

confidence: 99%

Assessment in textbooks: Exploring cognitive demand in first-year accounting textbooks

Ngwenya

Arek-Bawa

2020

SAJHE

View full text Add to dashboard Cite

The South African higher education program expects to develop graduates with high cognitive skills. This is endorsed further by the expectations of high cognitive attributes for entry level members from the Accounting professional bodies. Although learning is largely driven by assessment, relatively few studies have investigated assessments in Accounting textbooks. Furthermore, most used Bloom's taxonomy as their sole analytical lens. Using a Revised Bloom's taxonomy and Leong's levels of difficulty hybrid framework, this article examines the cognitive challenge of assessment tasks in selected level-one Financial Accounting textbooks used in South Africa and Kenya. The findings reveal the dominance of the Apply-Task-Hybrid and the possibility of enabling higher cognitive attributes within lower levels of cognition. The article provides new insight for accounting academics who seek to enhance their pedagogical engagement, as well as textbook producers, curriculum developers and professional accounting bodies. It also offers the Layered Analytical Framework as an alternative approach to examining cognitive demand across diverse fields of study, but specifically within the discipline of Accounting.

show abstract

Sources of difficulty in assessment: example of PISA science items

Cited by 24 publications

References 23 publications

Students' attitudes toward science and science achievement: An analysis of the differential effects of science instructional practices

Students' attitudes toward science and science achievement: An analysis of the differential effects of science instructional practices

The Semiotics of Test Design: Conceptual Framework on Optimal Item Features in Educational Assessment Across Cultural Groups, Countries, and Languages

Assessment in textbooks: Exploring cognitive demand in first-year accounting textbooks

Contact Info

Product

Resources

About