The Challenges of Measuring Outside-School-Time Educational Activities: Experiences and Lessons from the Programme for International Student Assessment (PISA)

Comparability of teachers’ educational background items in TIMSS: a case from Turkey

“…To collect information that can be comparable across countries, background surveys are developed in one or two languages and then the original items are adapted to different cultures and languages (Ebbs and Korsnakova 2015). The international studies put great effort into ensuring equivalent adaptations of background items, however there are cases where major differences in adaptations occur which result in variations in responses across countries or cultures (Bray et al 2020;Sjoberg 2015). On the other hand, researchers may have the tendency to taking the comparability of the background items as granted (Avvisati et al 2019;Bray et al 2020).…”

“…The international studies put great effort into ensuring equivalent adaptations of background items, however there are cases where major differences in adaptations occur which result in variations in responses across countries or cultures (Bray et al 2020;Sjoberg 2015). On the other hand, researchers may have the tendency to taking the comparability of the background items as granted (Avvisati et al 2019;Bray et al 2020). This paper presents one case from TIMSS in which survey question choices and responses highlight items that are "lost in translation", thus raising at least some questions about the broader validity for making high-stakes decisions.…”

“…Although there are multiple papers that discuss these adaptation problems, very few of them used actual data from large scale international studies to detect possible interpretation differences across different versions of tests or surveys (e.g. Arffman 2010Arffman , 2012Bray and Kobakhidze 2014;Bray et al 2020;Ercikan 1998). Moreover, most of these studies focus only on translation of items developed to measure a psychometric construct (Arffman 2010(Arffman , 2012Ercikan 1998).…”

“…Moreover, most of these studies focus only on translation of items developed to measure a psychometric construct (Arffman 2010(Arffman , 2012Ercikan 1998). There are very few studies that investigate comparability of background questionnaire items and open the validity of the information collected through these background items to discussion (see Bray and Kobakhidze 2014;Bray et al 2020).…”

“…Drawing from Bray et al (2020) recent study that showed major adaptation differences in different versions of items related to students' outside-of-school educational activities in PISA that resulted in variations in interpretations of these items across countries, this paper presents a case from TIMSS where differences in meaning between the Turkish and English versions of a teacher questionnaire item in TIMSS resulted in inconsistent responses from the participating science and mathematics teachers in Turkey. Complementing the findings of Bray et al (2020), this paper contributes to the discussion related to item adaptations of background questionnaire items by providing an example where translation issues may result in differences in interpretations of a background questionnaire item in TIMSS. This paper also contributes to the discussions related to "data literacy" of secondary data users by providing evidence for the existence of translation issues even for the items that are designed to measure "objective traits" of participants (Avvisati et al 2019;Bray et al 2020).…”