The Relative Validity and Usefulness of Various Empirical Research Designs

Abstract

Phenomena of interest to researchers in industrial and organizational (I-O) psychology and allied fields (e.g., human resource management, organizational behavior, organization theory) can be studied through research that uses a wide variety of design options. In this chapter, research design is defined as a unique combination of research setting and research strategy. The research design concept is defined somewhat differently in other works (e.g., Kerlinger, 1986;Runkel and McGrath, 1972;Stone, 1978). For example, Kerlinger defines research design as "the plan and structure of investigation so conceived as to obtain answers to research questions" (1986, p. 279); Runkel and McGrath note that research design is "a plan by which you will be able to reason, step by step, from the observations you [the researcher] intend to make to logically sound conclusions about problems or questions you are trying to resolve" (1972, p. 36). Rosenthal and Rosnow define design as "a blueprint that provides the scientist with a detailed outline or plan for the collection and analysis of data" (1991, p. 69).In any given study, the researcher must choose from among options concerning such factors as (a) research strategy (e.g., true experiment, quasi-experiment, non-experiment), (b) research setting (e.g., laboratory, field), research participants (e.g., students, workers, unemployed individuals), and (c) operational definitions of independent and dependent variables. The choices that are made about these factors influence the overall validity of conclusions that stem from a study Campbell, 1976, 1979;Cook, Campbell, and Peracchio, 1990;Kerlinger, 1986;Stone, 1978). As noted below, the overall confidence that a researcher has about the results of a study are a function of the validity of inferences about cause-effect relationships (internal validity), the correctness of statistical inferences derived from a study (statistical conclusion validity), the extent to which

The Relative Validity of Inferences About Mediation as a Function of Research Design Characteristics

“…Prior to describing the three above-noted types of experimental designs, it is important to distinguish between experimental designs and research settings (Stone-Romero, 2002, 2006c. Thus, in this section we focus on two setting-related issues, i.e., the laboratory versus field distinction and the influence of setting type on internal validity.…”

“…As such, inferences about mediation hinge on the internal validity of a study (Shadish et al, 2002;Stone-Romero & Rosopa, 2004). In view of this, we next consider the validity of inferences about causal relations between variables (e.g., as in mediation models) that result from three general types of experimental designs, i.e., nonexperimental, quasi-experimental, and randomized experimental (Shadish et al, 2002;Stone, 1978;Stone-Romero, 2002, 2006a, 2006b, 2006c.…”

Modelling Dynamic Affective and Decision Making Processes during Approach and Avoidance Goal Striving

“…There is also consistent evidence indicating that laboratory-based studies can have as much external validity as field studies when appropriate research designs and manipulations are used (C. A. Anderson & Bushman, 1997;Stone-Romero, 2004). In actuality, laboratory research provide the necessary level of control to develop and test psychological theories, particularly when complex and dynamic processes are involved.…”

The State of our Toolbox: A Meta-analysis of Reliability Measurement Precision

“…The resulting choices often involve considerations of feasibility, cost, duration, and adherence to ethical standards. Moreover, because there are inherent benefits and limitations within different methods, I/O researchers have to resolve the dilemmas that inevitably come with the methodological decisions they make (Stone-Romero, 2002). For example, choosing to conduct a lab experiment over a field study or using a questionnaire that was adapted and translated from another culture will provide advantages and disadvantages for certain types of validity.…”

“…As mentioned in the introduction, methodological decisions inevitably impact study limitations (Stone-Romero, 2002). Using five year intervals allowed for a longer period of analysis dating back to 1975 in order to capture long-term changes in reliability and journal content.…”