Incorporating Sampling Weights Into The Generalizability Theory For Large-Scale Analyses

Abstract

Large scale studies frequently use complex sampling procedures, disproportionate sampling weights, and adjustment techniques to account for potential bias due to nonresponses and to ensure that results from the sample can be generalized to a larger population. Survey researchers are concerned about measurement error and the use of weights in developing models. Consequently, multiple weighting factors are used and these weighting factors are manifested as a final survey (composite) weight available for analysis. We developed a method to incorporate an external weighting factor like this for analyses of measurement errors in the theory of generalizability to provide researchers with a tool to evaluate the measurement error components of survey quality and undesirable error components of large-scale assessment programs such as national and state assessments.

Measurement Considerations of Peak Stepping Cadence Measures Using National Health and Nutrition Examination Survey 2005–2006

“…Our findings did not take the complex sampling design of the NHANES into account due to the analytical complexity of variance component estimation in G-theory analyses. We are aware of a few studies that demonstrated methods of incorporating survey weights into the framework of G-theory; 28 however, to the best of our knowledge, no statistical software is currently available to accomplish such a goal. The G-theory framework assumes simple random sampling and therefore, our findings could likely be biased due to the disproportional sampling techniques…”

SEER: A Graphical Tool for Multidimensional and Categorical Data

“…Every two years, over 30,000 respondents are surveyed for the SDR. A considerable number of respondents are surveyed repeatedly (Chiu and Fecso, 2003). This type of panel data can answer questions of interest to policy makers in the federal government and academia such as: How do the occupation(s) of doctorate recipients change over time?…”