A Review and Analysis of the Policy-Capturing Methodology in Organizational Research: Guidelines for Research and Practice

Abstract

Policy-capturing has been employed extensively in the past to examine how organizational decision makers use the information available to them when making evaluative judgments. The purpose of this article is to provide researchers with guidelines for enhancing the reliability and validity of their studies. More specifically, the authors identify issues researchers may want to consider when designing such studies and offer suggestions for effectively addressing them. They draw on a review of 37 articles from 5 major journals to identify “best practice” and discuss the advantages and disadvantages of alternative approaches to resolving the various issues. The key issues are (a) the realism of the approach and its effect on both internal and external validity, (b) the limits of the full factorial design, (c) the need for orthogonal cues, (d) sample size and statistical power, and (e) the assessment of reliability. The analysis also includes comparisons with conjoint analysis, a similar methodology used in the marketing research literature.

Values and judgment under uncertainty: evidence from venture capitalist assessments of founders

“…Because there are many factors that infl uence outcomes in the context of VC decision making (e.g., Hall and Hofer, 1993;MacMillan et al, 1985;MacMillan et al, 1987;Sexton and BowmanUpton, 1990;Shepherd, 1999;Tyebjee and Bruno, 1984;Zacharakis and Meyer, 1998), designing our task around decision scenarios allowed us to control for (eliminate) factors that were not germane to our specifi c research questions about the role decision makers' values play in their evaluations of founders. Also, providing information on actual opportunities has advantages over other approaches that simply ask individuals to rate or rank variables of interest in order of importance; simple ranking approaches are subject to concerns about validity because stated policy decisions may differ from actual policies (e.g., Karren and Barringer, 2002;Sherer et al, 1987;Zacharakis and Meyer, 1998). In addition, our approach also follows the precedent of other studies on VC and strategic decision making where similar designs were used to better control the study context (e.g., Frederickson, 1984;Frederickson and Iaquinto, 1989;Hitt and Tyler, 1991;Hodgkinson et al, 1999).…”

Consumer perceptions of co-branding alliances: Organizational dissimilarity signals and brand fit

“…Each scenario represented a unique combination of different dimensions (Atzmüller and Steiner, 2010). We used a full factorial design (Karren and Barringer, 2002). The scenario universe of this study representing all combinations of four dimensions' variations comprised 16 scenarios.…”

“…The variables were all defined by two levels representing either high or low (or no, respectively) co-branding partner dissimilarity. An irregular number of levels may have distorted results (Karren and Barringer, 2002).…”

Prosocial knowledge mediates effects of agreeableness and emotional intelligence on prosocial behavior

“…For instance, someone who is naturally disagreeable may learn over time that expressing disagreeableness when trying to convince someone to do a favor may not be the most effective course of action in that particular situation. In the same way individuals form policies in the policy capturing literature (Karren & Barringer, 2002), ITP theory proposes that people form policies about the effectiveness of trait expression. Thus, ITPs can denote knowledge about effective trait expression when they are aligned with the beliefs of experts.…”