Thematic Content Analysis of Studies Using Generalizability Theory

Abstract

One of the important theories in education and psychology is Generalizability (G) Theory and various properties distinguish it from the other measurement theories. To better understand methodological trends of G theory, a thematic content analysis was conducted. This study analyzes the studies using generalizability theory in the field of education in Turkey by using the method of thematic content analysis. It reviews 60 studies, including 31 articles and 29 theses published from 2004 to 2017. The selected studies underwent thematic content analysis using parameters including tagged information, aim, G Theory type, number of facets used in the study, Turkish word for "facet," object of measurement, sample size, design type, mixed-design availability, shared results of G and D studies, computer programs, method of calculating negative variance, availability of fixed facets, and design balance. The data were interpreted on the basis of frequencies; both table and figures are included in the study. According to the results, there is an increase in the number of studies conducted by using G theory by years. Of these, many compare theories; most of them applying univariate G Theory and consider two-faceted measurement situations. While a small subset of studies features mixed design, a large group features crossed design, with individuals as the object of measurement. The computer program most commonly used in analyses is EduG. The majority of studies use balanced design. Recommendations are provided accordingly with the results.

The thematic content analysis of the scales used in citizenship education

“…The checklist aimed to set standard criteria for content analysis of the scales and consisted of two sections called "study tag" and "theoretical information". When we look at the content analysis studies in the literature (Kaya Uyanık et al, 2017;Taşdelen Teker & Güler, 2019), it is seen that in the case of using a checklist, expert opinion is sought to determine the suitability of the checklist in terms of scope and content. From the point of this view, two experts of measurement and evaluation who studied scale development and scale adaptation were consulted for their opinions of the checklist.…”

Investigation of the Reliability of Teachers, Self and Peer Assessments at Primary School Level with Generalizability Theory

“…In addition to his; in G theory, the sources of error can be addressed separately and the interactions of error sources can be determined as a result of the analysis (Brennan, 2001;Güler, 2009;Güler, Kaya-Uyanık & Taşdelen-Teker, 2012;Shavelson & Webb, 1991). The study carried out by Taşdelen-Teker and Güler (2019) shows that G Theory is frequently used especially in inter-rater reliability and standard-setting studies. Due to these advantages and application areas of G Theory, in this study, G Theory was preferred in order to determine the reliability between different types of raters (teacher-self-peer).…”

Coping with Unbalanced Designs of Generalizability Theory: G String V

“…For practical reasons, it cannot handle more than four stratification facets. According to the results of two review studies conducted by Rios, Li and Faulkner-Bond (2012) and Taşdelen Teker and Güler (2019), there is no study that has more than one stratification facet. Meanwhile, when a researcher has more than four stratification facets it has been suggested to collapse the facets that are unlikely to contribute to error variance.…”

“…One other reason for unbalanced situations can be missing observations from crossed and nested designs (Webb, Shavelson & Haertel, 2006). As sample sizes tend to be small in the G theory analyses (Rios, Li & Faulkner-Bond, 2012;Taşdelen Teker & Güler, 2019), missing data becomes an important topic. Researchers normally prefer listwise deleting, inputing missing observations, or employing unbalanced designs to deal with missing data.…”

“…Taşdelen Teker and Güler (2019) conducted a thematic content analysis of studies using the G theory in the field of education in Turkey and found that 6 of 60 studies were conducted with unbalanced design. According to the results of the above-mentioned review studies of Rios, Li and Faulkner-Bond (2012), and Taşdelen Teker and Güler (2019), the ratio of unbalanced designs is high enough to make coping with them indispensable.…”