The Effects of Range Restriction on Estimates of Criterion Interrater Reliability: Implications for Validation Research

Sackett, Paul R.; Laczo, Roxanne M.; Arvey, Richard D.

doi:10.1111/j.1744-6570.2002.tb00130.x

Cited by 44 publications

(60 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further inspection of scatter plot distributions of test vs. retest for negative and positive mood variables illustrated that low correlations were largely a function of skewed distributions, resulting in range restriction. 35 Participants tended to endorse extreme values (0 or 100) for both test and retest, thus showing high reliability, but low correlations because of range restriction. This also explains why the mood ratings showed some validity (difference between conditions) despite low calculated reliability.…”

Section: Discussionmentioning

confidence: 98%

Analgesic Placebo Treatment Perceptions: Acceptability, Efficacy, and Knowledge

Kisaalita

2012

The Journal of Pain

View full text Add to dashboard Cite

Current evidence supports the efficacy of placebo analgesia and illustrates that patients may be open to placebo use despite uncertainty regarding its mechanisms. Debate persists, however, concerning the ethics of placebo treatments. The purpose of the present web-based study was to expand upon the empirical literature on placebo analgesia ethics and acceptability. Participants (n = 100) provided their definition of a placebo and responded to 24 questions addressing placebo analgesia perceived knowledge, acceptability, effectiveness, and likelihood of placebo use among different healthcare providers. Results support previous research on the effects of placebo on negative mood and healthcare provider attributions, with findings illustrating that negative consequences of administration were largely mitigated by a beneficial treatment outcome. Results showed that participants conceptualized placebo as predominately “inert” and were mixed regarding interpretations of placebo effectiveness. Though acceptability ratings were dependent on the context of placebo administration, participants endorsing even moderate placebo acceptability were more open to placebo interventions and reported overall more positive treatment outcomes. Participants believed that placebos were used differentially among healthcare providers. Additional studies are needed to determine if placebo education can beneficially impact perceptions of placebo analgesia knowledge, acceptability, and treatment effectiveness.

show abstract

Section: Discussionmentioning

confidence: 98%

Analgesic Placebo Treatment Perceptions: Acceptability, Efficacy, and Knowledge

Kisaalita

2012

The Journal of Pain

View full text Add to dashboard Cite

show abstract

“…Given that this institution does not have an open-admissions model, a certain restriction of range is observed among students: average Math SAT scores of this student sample (see Table 1) suggest that this is a slightly more able group than a randomly drawn national sample would be. Given that restriction of range can impact reliability results (Sackett, Laczo, & Arvey, 2002), the correlations and subsequent path coefficients could be attenuated (Roth et al, 2014). Replicating the same analyses using different student samples is recommended.…”

Section: Limitations and Future Studiesmentioning

confidence: 98%

The Role of Students’ Attitudes and Test-Taking Motivation on the Validity of College Institutional Accountability Tests: A Path Analytic Model

Zilberberg

Finney

Marsh

et al. 2014

International Journal of Testing

View full text Add to dashboard Cite

Given worldwide prevalence of low-stakes testing for monitoring educational quality and students' progress through school (e.g., Trends in International Mathematics and Science Study, Program for International Student Assessment), interpretability of resulting test scores is of global concern. The nonconsequential nature of low-stakes tests can undermine students' test-taking motivation, artificially deflating performance and thus jeopardizing validity of test-based inferences, whether they pertain to programs, institutions, or nations (Eklöf, 2007(Eklöf, , 2010Stanat & Lüdtke, 2013;Wise & DeMars, 2005). Moreover, students in countries such as the United States, where academic progress over the course of K-12 (kindergarten through Grade 12) is systematically assessed, are likely to develop antagonistic attitudes toward lowstakes testing by the time they enter college. The relationship between such attitudes, test-taking motivation, and performance on a low-stakes university accountability test was modeled via path analysis. Results indicated the effects of attitudes were indirect (via test-taking motivation) and minimal, suggesting the influence of attitudes on test performance is negligible, further supporting the validity of inferences made from such low-stakes tests. Implications for international assessment are discussed.

show abstract

“…Although this may be the least-biased estimate of interrater reliability available, there is debate about the appropriateness of using interrater correlations to estimate reliability (see Murphy & DeShon, 2000;Schmidt, Viswesvaran, & Ones, 2000). More recently, Sackett, Laczo, and Arvey (2002) showed a tendency for interrater reliability corrections to produce overestimates in validity coefficients when such estimates are not initially corrected for range restriction. To test the possibility that our estimates linking each job attitude to job performance were inflated, we re-ran our theorized models using recalculated meta-analytic estimates based on hypothetical interrater reliability estimates for job performance of .80.…”

Section: Limitations and Research Directionsmentioning

confidence: 99%

How Important are Job Attitudes? Meta-Analytic Comparisons of Integrative Behavioral Outcomes and Time Sequences

Harrison

Newman

Roth

2006

AMJ

1,021

900

View full text Add to dashboard Cite

Drawing on the compatibility principle in attitude theory, we propose that overall job attitude (job satisfaction and organizational commitment) provides increasingly powerful prediction of more integrative behavioral criteria (focal performance, contextual performance, lateness, absence, and turnover combined). The principle was sustained by a combination of meta-analysis and structural equations showing better fit of unified versus diversified models of meta-analytic correlations between those criteria. Overall job attitude strongly predicted a higher-order behavioral construct, defined as desirable contributions made to one's work role (r ‫؍‬ .59). Time-lagged data also supported this unified, attitude-engagement model.Job attitudes and job performance are perhaps the two most central and enduring sets of constructs in individual-level organizational research. Yet, a longstanding debate persists about the nature and the strength of relationships between these fundamental predictors and criteria (Austin & Villanova, 1992;Brief, 1998;Johns, 1998;Judge, Thoreson, Bono, & Patton, 2001). An elemental question remains: How important are job attitudes for predicting and understanding job performance in particular, and work role-directed behaviors in general?Authors of early qualitative reviews concluded that only weak support existed for the relationship between one principal attitude, job satisfaction, and supervisor ratings or output measures of job performance (e.g., Brayfield & Crockett, 1955). A common inference in those reviews was that job attitudes were more strongly related to absence, turnover, and other forms of work role withdrawal than they were to in-role performance (e.g., Herzberg, Mausner, Peterson, & Capwell, 1957;Vroom, 1964). Subsequent quantitative reviews also failed to show job attitudes as having strong predictive utility. One meta-analysis reported a lackluster value (ˆϭ .17) as the best estimate of the correlation between satisfaction and performance (Iaffaldano & Muchinsky, 1985). Another review showed organizational commitment bore a weaker relationship to job performance (ˆϭ .14) than to at least one withdrawal behavior, turnover (ˆϭ Ϫ.28;Mathieu & Zajac, 1990). Consequently, the pendulum of causal potency has swung away from job attitudes (at least until recently; see Judge et al. [2001]). One widely held view is that attitudes are inconsistent or epiphenomenal forces in work behavior (e.g., Locke & Latham, 1990): they explain only 3-4 percent of performance variance and have little practical importance for managers.The current article subjects that view to empirical scrutiny via comprehensive and comparative tests. In doing so, we attempt to contribute to management knowledge in five ways. First, we investigate and more fully map the individual-level criterion space (i.e., a set of work behaviors valued by organizations [Austin & Villanova, 1992]) by bringing four original meta-analyses to the literature, estimating the connections between contextual performance and (1) lateness, (2) a...

show abstract

The Effects of Range Restriction on Estimates of Criterion Interrater Reliability: Implications for Validation Research

Cited by 44 publications

References 13 publications

Analgesic Placebo Treatment Perceptions: Acceptability, Efficacy, and Knowledge

Analgesic Placebo Treatment Perceptions: Acceptability, Efficacy, and Knowledge

The Role of Students’ Attitudes and Test-Taking Motivation on the Validity of College Institutional Accountability Tests: A Path Analytic Model

How Important are Job Attitudes? Meta-Analytic Comparisons of Integrative Behavioral Outcomes and Time Sequences

Contact Info

Product

Resources

About