At Sea With Synthetic Validity

Steel, Piers; Johnson, Jeff W.; Jeanneret, P. Richard; Scherbaum, Charles A.; Hoffman, Calvin C.; Foster, Jeff

doi:10.1111/j.1754-9434.2010.01255.x

Cited by 6 publications

(3 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Vocational counselors might well take note that the traits associated with occupational interest do not necessarily translate into those associated with occupational attainment, as per Lubinski and Benbow (). Also, Steel et al () argue, the future of selection will be in assessing personality facets, like procrastination. When considering narrow as well as broad performance dimensions, facet level analyses are often preferable (Dudley et al, ; Schneider, Hough, & Dunnette, ).…”

Section: Discussionmentioning

confidence: 99%

Procrastination's Impact in the Workplace and the Workplace's Impact on Procrastination

Nguyen

Steel

Ferrari

2013

Int J Selection Assessment

Self Cite

125

View full text Add to dashboard Cite

Procrastination is a self-regulatory failure, whose costs are debated. Here, we establish its impact in the workplace. Using an Internet sample, we assessed 22,053 individuals in terms of their sex, employment status, employment duration, income, occupational attainment and level of procrastination. High levels of procrastination is associated with lower salaries, shorter durations of employment, and a greater likelihood of being unemployed or under employed rather than working full-time. Also, procrastination partially mediates sex's relationship with these work variables. Women tend to procrastinate less than men, evidently giving women an employment advantage. If women procrastinated the same as men, there should be 1.5 million fewer women in full-time employment in the US. alone. Determining the causes of procrastination in the workplace, we also examined it at an occupational level. The results strongly support the gravitational hypothesis: jobs that require higher levels of motivational skills are less likely to retain procrastinators. However, there was some support that jobs can foster procrastination. Procrastinators tend to have jobs that are lower in intrinsically rewarding qualities.

show abstract

Section: Discussionmentioning

confidence: 99%

Procrastination's Impact in the Workplace and the Workplace's Impact on Procrastination

Nguyen

Steel

Ferrari

2013

Int J Selection Assessment

Self Cite

125

View full text Add to dashboard Cite

show abstract

“…If correct, as Murphy (2009) notes, there is little need for synthetic validity, which tries to account for the variance among validity coefficients by use of job analyses, and the topic has been settled. However, prior work documents how this near homogeneity seen for job performance validity coefficients is largely artificial, the result of a series of statistical and methodological errors (Steel, Johnson, Jeanneret, Scherbaum, Hoffman, & Foster, 2010). By reexploring the issue of situational moderators at the occupational level, Steel and Kammeyer-Mueller (2009) accounted for 25 times more variance than the original moderator analyses, which used less fine-grained job description information.…”

Section: Discussionmentioning

confidence: 98%

Improving the Meta-Analytic Assessment of Effect Size Variance With an Informed Bayesian Prior

Steel

Kammeyer‐Mueller

Paterson

2014

Journal of Management

Self Cite

View full text Add to dashboard Cite

Meta-analytic estimation of effect size variance is critical for determining the degree to which a relationship or finding generalizes across contexts. In most meta-analyses, population effect size variability is estimated by subtracting expected sampling error variance from observed variance, using only information from a limited set of available studies. We propose an improved Bayesian variance estimation technique that incorporates findings from previous meta-analytic research through an informed prior distribution of likely levels of effect size variance. The logic of exchangeability as a conceptual foundation for using an informed prior is explicated. On the basis of Monte Carlo simulations, we find the traditional method of meta-analytic variance estimation the most biased and least accurate technique across all sizes of meta-analyses considered. The Bayesian methodology incorporating an informed prior proved to be the most accurate and overall least biased of all estimation methods. Conceptual advantages and limitations that must be taken into account when incorporating an informed prior to estimate variability of effect sizes in a meta-analysis are also discussed.

show abstract

“…example, combining moderator search with meta-analytic structural equation modeling (MASEM) allows us to predict validity coefficients without a local validation study and build instantly personnel selection systems that are costless and of high quality (Steel et al, 2010).…”

mentioning

confidence: 99%

Leftovers and boundary conditions: a moderator proposal

Steel

2023

Front. Organ. Psychol

Self Cite

View full text Add to dashboard Cite

Systematic reviews and meta-analyses are a chance for a field to take stock and see what is now known (Hunt, 1997). Articles are searched for, then gathered, and then coded into a common metric, allowing for statistical analysis. Inevitably, for any meta-analysis of a decent size, the effect sizes from all these studies are not the same: variation around the mean is the norm . Part of this is due to sampling error, where the random chance draw of participants or data points accounts for fluctuations. However, even after accounting for the uncertainty associated with finite sample size, there is residual variance-the leftovers. Like culinary leftovers, they can seem like an afterthought, and some have argued they are of no great importance (LeBreton et al., 2017). However, accounting for them is foundational to the advancement of our science.This leftover variance goes by a variety of names, from tau to the REVC (Random Effects Variance Component). It reflects the possible outcomes or the sizes of your effect sizes, the range of which is called credibility or prediction intervals. Basically, if a study was redone within the confines of what was done before, it effects size should be within these intervals. Credibility intervals are often broad and can cross the correlational Rubicon of zero, where effect sizes fail to even directionally generalize. Accounting for this variation can be all too important. Without it, each of our studies is simply a snapshot frozen in time, speaking to what happened in that particular moment in that specific setting, which may or may not happen again or at least not to the same extent (Yarkoni, 2022). On the other hand, if we can identify the sources of variation, we have a pathway to a mature science that can make precise predictions based on diagnosis or assessment alone. It can tell you when and where a finding is applicable and to who. For example, a medical treatment might cure some and kill others, so best if we could predict that variation. Unfortunately, we often can't. In the sobering words of Flake et al. ( 2022): "any statistical model estimated from any study has so many omitted sources of variance that the estimates are likely meaningless" (p. 33).When a meta-analysis is conducted, a moderator analysis tries to account for this variation in effect sizes. Along these lines, we have advanced statistical methodology quite far, moving from simply subgroup analysis, where we compare the effect sizes of two group and see if one is bigger, to souped-up multiple regression schemes using continuous variables and sophisticated weighting (Steel et al., 2021). Despite these statistical refinements, often meta-analysts find that the moderators they want are not in the literature obtained . Aside from type of measure, studies typically confine their reporting to the thin gruel of participant age, gender ratio, student status, and nation. If you see an abundance of meta-analyses that use culture as a moderator, well there is a reason. Despite meta-analyses being invaluable summaries, a...

show abstract

At Sea With Synthetic Validity

Cited by 6 publications

References 53 publications

Procrastination's Impact in the Workplace and the Workplace's Impact on Procrastination

Procrastination's Impact in the Workplace and the Workplace's Impact on Procrastination

Improving the Meta-Analytic Assessment of Effect Size Variance With an Informed Bayesian Prior

Leftovers and boundary conditions: a moderator proposal

Contact Info

Product

Resources

About