Assessing Differential Prediction of College Grades by Race/Ethnicity with a Multilevel Model

Culpepper, Steven Andrew; Davenport, Ernest C.

doi:10.1111/j.1745-3984.2009.00079.x

Cited by 24 publications

(21 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There are many topics and fields that make substantial use of moderation including research examining aptitude-by-treatment interactions (DeShon & Alexander, 1996;Dretzke, Levin, & Serlin, 1982), gender difference studies , differential prediction studies (Culpepper & Davenport, 2009;DeShon & Alexander, 1996), differential validity studies (Linn, 1987), industrial/organizational psychology (Champoux & Peters, 1987), and the job design literature (Champoux & Peters, 1987). It may be easier for researchers in fields with more mature bodies of literature to hypothesize about complex relationships such as moderation.…”

Section: Moderated Multiple Regression (Mmr)mentioning

confidence: 99%

“…MMR methods are also used in examining differential validity and differential prediction (Culpepper & Davenport, 2009;Linn, 1978). Differential validity assesses whether the magnitude of association between IVs and DVs (for example, between SAT score and college GPA) differs by subgroup and is typically assessed by looking for significant differences in correlation coefficients by group.…”

Section: Moderated Multiple Regression (Mmr)mentioning

confidence: 99%

“…Differential prediction examines whether a single regression equation is appropriate for all subgroups (Mattern, Patterson, Shaw, Kobrin, & Barbuti, 2008). For example, one differential prediction study (Culpepper & Davenport, 2009) used MMR to find that the relationship between high school achievement measures (high school GPA and SAT scores) and college GPA differed by race/ethnicity. Specifically, given the same level of high school achievement, African American students are predicted to have lower average college achievement compared with White students.…”

Section: Moderated Multiple Regression (Mmr)mentioning

confidence: 99%

“…In other words, high school achievement (IV) interacts with race/ethnicity (MV) in predicting college GPA (DV) meaning that one prediction equation for all racial/ethnic groups would not be appropriate. Understanding these prediction equations is critical since postsecondary institutions use them for high-stakes judgments, such as admission decisions (Culpepper & Davenport, 2009). …”

Section: Moderated Multiple Regression (Mmr)mentioning

confidence: 99%

“…In order to use the Wald test to test for a significant interaction, the researcher can assess significance of B 3 , which is the slope coefficient for the product term in the full model (Aiken & West, 1991;Baron & Kenny, 1986;Jose, 2013). This method is suggested by some researchers specifically addressing moderation (Aiken & West, 1991;Baron & Kenny, 1986;Jose, 2013) and has been used in substantive research for assessing interaction effects within multilevel models (Culpepper & Davenport, 2009). …”

Section: Wald Testmentioning

confidence: 99%

See 4 more Smart Citations

Estimating individual-level interaction effects in multilevel models: a Monte Carlo simulation study with application

Lorah

2017

Journal of Applied Statistics

View full text Add to dashboard Cite

Moderated multiple regression (MMR) provides a useful framework for understanding moderator variables. When the relationship between an independent variable (predictor) and a dependent variable (criterion) varies as a function of a third variable, that third variable is considered a moderator of the predictor-criterion relationship. Moderated relationships can also be examined within datasets that include nesting, for example individuals might be nested within groups. However, the literature is not clear on the best way to assess data for significant moderating effects, particularly within a multilevel modeling framework. This study explores potential ways to test moderation at the individual level (level 1) within a 2-level multilevel modeling framework, with varying effect sizes, cluster sizes, and numbers of clusters that represent realistic conditions in applied educational research. The study examines five potential methods for testing interaction effects: the Wald test, F-test, likelihood ratio test, BIC, and AIC.For each method, the simulation study examines how Type I error rates vary as a function of number of clusters and cluster size and how power varies as a function of number of clusters, cluster size, and interaction effect size. Following the simulation study, an applied study uses real data to assess interaction effects using the same five methods. Results indicate that the Wald test, F-test, and likelihood ratio test all perform similarly in terms of Type I error rates and power. Type I error rates for the AIC are more liberal, and for the BIC typically more conservative. A four-step procedure for applied researchers interested in examining interaction effects in multi-level models is provided. gender is present. Further, the data for this study (Litzler et al., 2014) included students nested within multiple schools, indicating that a multilevel model would be appropriate, given the non-independence of individual student observations. Moderator hypotheses such as these can be examined with moderated multiple regression models and this can be done within a multilevel framework for data that includes nesting. The results of these types of models can represent important and relevant contributions to the literature. This study will investigate methods for testing multilevel moderated regression models with interaction effects at the individual level only (or more generally, at level-one only). Within this study, this model includes a continuous dependent variable, a continuous independent variable, and a binary moderator variable. Methods for testing this interaction effect will be evaluated on Type I error rates and power based on a simulation study and an applied study will follow to illustrate the technique. An introduction to moderation, multilevel models, and testing for interaction effects follows in this chapter.

show abstract

Section: Moderated Multiple Regression (Mmr)mentioning

confidence: 99%

Section: Moderated Multiple Regression (Mmr)mentioning

confidence: 99%

Section: Moderated Multiple Regression (Mmr)mentioning

confidence: 99%

Section: Moderated Multiple Regression (Mmr)mentioning

confidence: 99%

Section: Wald Testmentioning

confidence: 99%

See 3 more Smart Citations

Estimating individual-level interaction effects in multilevel models: a Monte Carlo simulation study with application

Lorah

2017

Journal of Applied Statistics

View full text Add to dashboard Cite

show abstract

Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex‐ante and ex‐post peer evaluation data from the Austrian science fund

Mutz

Bornmann

Daniel

2014

Asso for Info Science & Tech

View full text Add to dashboard Cite

It is essential for research funding organizations to ensure both the validity and fairness of the grant approval procedure. The ex-ante peer evaluation (EXANTE) of N = 8,496 grant applications submitted to the Austrian Science Fund from 1999 to 2009 was statistically analyzed. For 1,689 funded research projects an ex-post peer evaluation (EXPOST) was also available; for the rest of the grant applications a multilevel missing data imputation approach was used to consider verification bias for the first time in peer-review research. Without imputation, the predictive validity of EXANTE was low (r = .26) but underestimated due to verification bias, and with imputation it was r = .49. That is, the decision-making procedure is capable of selecting the best research proposals for funding. In the EXANTE there were several potential biases (e.g., gender). With respect to the EXPOST there was only one real bias (discipline-specific and year-specific differential prediction). The novelty of this contribution is, first, the combining of theoretical concepts of validity and fairness with a missing data imputation approach to correct for verification bias and, second, multilevel modeling to test peer review-based funding decisions for both validity and fairness in terms of potential and real biases.It is important for a research funding organization to ensure the quality of its peer review-based decision-making procedure. On behalf of the Austrian Science Fund (FWF), Austria's leading funding organization for basic research, external meta-evaluations of its grant approval procedures were recurrently conducted (Dinges, 2005). For a metaevaluation of a research funding organization's decisionmaking procedure, the following three questions are essential (Bornmann & Daniel, 2005;Daniel, 2004): (a) Is the procedure reliable-that is, what is the extent of agreement among reviewers on the quality of a grant application and on whether the proposed research project should be funded or not (ex-ante peer evaluation, EXANTE)?; (b) Is the procedure valid-that is, do peer reviewers' overall ratings (EXANTE) or the funding decisions of the Board of Trustees correlate with scientific performance measures subsequent to the funding decisions (ex-post peer evaluation, EXPOST)? Is the research funding organization capable of selecting the best grant proposals (EXANTE) with the best performance for funding (EXPOST)? (Irvine & Martin, 1984;Van den Besselaar & Leydesdorff, 2009); Is the procedure fair-that is, are there external factors that have an effect on the decision-making procedure but that have nothing to do with the quality of a grant proposal or the performance of a research project?

show abstract

Assessment in Game-Based Learning

Ifenthaler

Eseryel

Ge³

2012

117

View full text Add to dashboard Cite

Assessing Differential Prediction of College Grades by Race/Ethnicity with a Multilevel Model

Cited by 24 publications

References 60 publications

Estimating individual-level interaction effects in multilevel models: a Monte Carlo simulation study with application

Estimating individual-level interaction effects in multilevel models: a Monte Carlo simulation study with application

Testing for the fairness and predictive validity of research funding decisions: A multilevel multiple imputation for missing data approach using ex‐ante and ex‐post peer evaluation data from the Austrian science fund

Assessment in Game-Based Learning

Contact Info

Product

Resources

About