An Extension and Field Test of the Retranslation Method for Developing Rating Scales

Kavanagh, Michael J.; Duffy, John F.

doi:10.1111/j.1744-6570.1978.tb00455.x

Cited by 14 publications

(13 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, a number of authors have raised questions about the techniques typically used to develop rating scales (Bernardin & Kane, 1980;Dickinson & Tice, 1977;Kane & Bernardin, 1982;Kavanagh & Duffy, 1978;Latham, Saari, & Fay, 1980;Schwab et al, 1975;Shapira & Shirom, 1980). The development of behavior-based rating scales starts with the generation of "critical incidents" of job-relevant behaviors and accomplishments.…”

mentioning

confidence: 99%

Job Performance Measurement in the Military: A Classification Scheme, Literature Review, and Directions for Research

Kavanagh¹,

Borman²,

Hedge³

et al. 1987

Self Cite

View full text Add to dashboard Cite

mentioning

confidence: 99%

Job Performance Measurement in the Military: A Classification Scheme, Literature Review, and Directions for Research

Kavanagh¹,

Borman²,

Hedge³

et al. 1987

Self Cite

View full text Add to dashboard Cite

“…As implied throughout, a performance evaluation system is beneficial if it can contribute to various aspects of organizational functioning in addition to that of performance appraisal. Kavanagh and Duffy (1978) illustrate a "spin-off' use of the retranslation method of scale development for the evaluation of an educational television series designed to teach basic reading skills.…”

Section: A Utilization Criteriamentioning

confidence: 99%

“…Thus, one suggestion is that the final set of BARS be determined after a factor analysis of obtained ratings identifies those items which survive retranslation but yet are still factorially complex; in other words, another stage is added to the development of BARS but this stage is based on preliminary utilization of the scales. Kavanagh and Duffy (1978) did adopt such a strategy and found that factor analysis of field data expanded the number of dimensions in scales which were used to evaluate a television series designed to teach basic reading skills.…”

Section: B Bars Utilization Studiesmentioning

confidence: 99%

Expectations of Behaviorally Anchored Rating Scales

Jacobs

Kafry

Zedeck

1980

Personnel Psychology

117

View full text Add to dashboard Cite

Behaviorally anchored rating scales (BARS) are reviewed from two perspectives. First, the particular BARS methodology is assessed on the basis of its fulfilling three broad categories of “criteria for criteria:” (1) utilization criteria, (2) qualitative criteria, and (3) quantitative or psychometric criteria. These three broad categories are composed of 14 specific criteria by which performance evaluation methodologies can be assessed. The second perspective involves an evaluation of BARS in terms of psychometric criteria but with the literature broken down into studies concerned with (1) development of BARS, (2) utilization of BARS, (3) comparison of BARS to other methods, and (4) rater training in BARS use. Conclusions from these reviews indicate that BARS is no better or worse than other methods when assessed on a quantitative basis whereas it has greater potential when assessed on the utilization and qualitative criteria. Suggestions are offered for extending BARS research to process questions and to domains other than performance appraisal perse

show abstract

“…Alternatively, when resources are scarce and fewer SME groups are available, the same SMEs may be called upon to perform multiple tasks; often this entails the same group of SMEs rating and retranslating critical incidents (Bernardin et al, 1976;Campbell et al, 1973;Hedge et al, 2004;Kavanagh & Duffy, 1978). The retranslation process may also be extremely truncated; in Campion, Pursell, and Brown (1988), an analyst independent of the project reclassified 9% of the critical incidents.…”

Section: Exploring Methods For Developing Barsmentioning

confidence: 99%

Exploring Methods for Developing Behaviorally Anchored Rating Scales for Evaluating Structured Interview Performance

Kell

Martín‐Raugh

Carney

et al. 2017

ETS Research Report Series

View full text Add to dashboard Cite

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Service.The Daniel Eignor Editorship is named in honor of Dr. Daniel R. Eignor, who from 2001 until 2011 served the Research and Development division as Editor for the ETS Research Report series. The Eignor Editorship has been created to recognize the pivotal leadership role that Dr. Eignor played in the research publication process at ETS. ETS Research Report Series ISSN 2330-8516 R E S E A R C H R E P O R T Exploring Methods for Developing Behaviorally Anchored Rating Scales for Evaluating Structured Interview PerformanceHarrison J. Kell, Michelle P. Martin-Raugh, Lauren M. Carney, Patricia A. Inglese, Lei Chen, & Gary FengEducational Testing Service, Princeton, NJ Behaviorally anchored rating scales (BARS) are an essential component of structured interviews. Use of BARS to evaluate interviewees' performance is associated with greater predictive validity and reliability and less bias. BARS are time-consuming and expensive to construct, however. This report explores the feasibility of gathering participants' responses to structured interview questions through an online crowdsourcing platform and using those responses to develop BARS. We describe the development of 12 structured interview questions to assess four applied social skills, elicitation of responses to these questions in the form of critical incidents from 68 respondents, and the creation of BARS from these critical incidents. Results indicate online participants are able to produce responses of sufficient quality to generate BARS for evaluating structured interview performance. We conclude by discussing limitations to this approach and future directions for research and practice.Keywords Amazon Mechanical Turk; behaviorally anchored rating scales; crowdsourcing; employment interviews; performance appraisal; social skills; structured interviews doi:10.1002/ets2.12152 Employment interviews are one of the most popular means of selecting personnel (Levashina, Hartwell, Morgeson, & Campion, 2014;McDaniel, Whetzel, Schmidt, & Maurer, 1994). Structured interviews that present all interviewees with the same standardized questions have higher validities for pre...

show abstract

An Extension and Field Test of the Retranslation Method for Developing Rating Scales

Cited by 14 publications

References 13 publications

Job Performance Measurement in the Military: A Classification Scheme, Literature Review, and Directions for Research

Job Performance Measurement in the Military: A Classification Scheme, Literature Review, and Directions for Research

Expectations of Behaviorally Anchored Rating Scales

Exploring Methods for Developing Behaviorally Anchored Rating Scales for Evaluating Structured Interview Performance

Contact Info

Product

Resources

About