How Many Students and Items Are Optimal for Teaching Level Evaluation of College Teachers? Evidence from Generalizability Theory and Lagrange Multiplier

Abstract

Budget and cost are two of the problems that cannot be ignored when conducting a measure study. Based on the application of generalizability theory, combined with Lagrange multiplier, this paper explores how many students and items are optimal for teaching level evaluation of college teachers under budget constraints to maintain the sustainable development of higher education. A total of 397 students are required to evaluate 10 teachers’ teaching level using the Teaching Level Evaluation Questionnaire for College Teachers, and we make different generalizability designs (i.e., (s:t) × i, (s:t) × (i:v) and (s:t) × (i:v) × o) for the collected data. The study unifies the Lagrange multiplier formula, derives the optimal sample size formula of different designs under budget constraints in generalizability theory, and calculates the optimal sample size for teaching level evaluation of college teachers in different designs with the estimated variance components. Results indicate that: (1) the unified formula of Lagrange multiplier has a stronger robustness and can be applied to different study designs under budget constraints in generalizability theory; (2) the occasion has a great effect on teaching level evaluation for college teachers; (3) the (s:t) × (i:v) × o design has a high efficiency in estimating the optimal sample size of teaching level evaluation for college teachers; (4) the design of (s:t) × (i:v) × o is the optimal generalizability design of teaching level evaluation for college teachers under budget constraints in generalizability theory; and (5) under budget constraints of teaching level evaluation for college teachers in generalizability theory, the optimal sample size of students is 31 for each teacher and the optimal sample size of items is 7 for each dimension.

Evaluation of Teaching Practice: Assessment of the Dimensionality of the Intern Teaching Evaluation Form (ITEF)

“…GT is a psychometric theory which is founded on a statistical sampling method, which in a one-shot analysis, has the ability to divide observed scores into their primary sources of variation. It must be noted that, the conception and estimation of reliability by classical test theory (CTT) is widened by GT through the modelling of a conceptual framework, founded in statistics, that allows an investigator to untangle multiple error sources that constitute the undistinguishable error (E) in CTT (Brennan, 2001;Li, 2022;Shavelson & Webb, 1991). Primarily, GT offers a framework that is used to detect and estimate the sources of errors of measurement in a measurement procedure, after which decisions can be made to improve it so as to give more reliable scores and at the same time economise the use of resources (Li et al, 2014).…”

Which method is optimal for estimating variance components and their variability in generalizability theory? evidence form a set of unified rules for bootstrap method

“…Generalizability theory (G theory) is widely used in psychological and educational measurement [1][2][3]. In G theory, estimating variance components (VC) is the essential technique and is of central importance.…”

Reliability of Students’ Teaching Practice Scores Using Generalizability Theory

“…This study fully used Generalisability Theory (GT), which is a psychometric model which is anchored on a statistical sampling technique which in a single analysis, partitions obtained scores into their primary sources of variation. GT provides a framework which is used to pinpoint and quantify the sources of measurement error based on which decisions can be taken to enhance the measurement procedure to give the most reliable scores (Li, 2022). Shavelson and Webb (1991), assert that, the Classical Test Theory (CTT) applies traditional approaches of estimating score reliability, and if applied, would estimate only a single source of measurement error at a time in a given analysis.…”

“…It is due to this major limitation of CTT in the estimation of reliability that GT comes in. The conception and estimation of reliability by CTT is broadened by GT through the provision of a conceptual framework which is grounded in statistics, that allows a researcher to untangle numerous sources of error that make up the undistinguishable error (E) in CTT (Li, 2022;Webb & Shavelson, 2005). This study is an addition to the literature on the application of GT in measurement instrument development, as a resource material for researchers in the social sciences, especially in the Ghanaian educational system and other developing countries where GT application is not pervasive (Shavelson & Webb, 1991).…”