Conducting General versus Population and Setting-Specific Meta Analyses

Abstract

Meta-analysis is an increasingly popular technique that systematically and statistically summarizes the results from research studies in a particular area (Lipsey & Wilson, 2001). Meta-analytic studies can be particularly useful to practitioners because they combine the results from various studies and eliminate the drudgery of sifting through the findings of multiple individual studies. Although meta-analysis can be used to summarize research in multiple areas (e.g., correlational studies), I am going to restrict my comments to intervention studies with a focus on career interventions. Specifically, this chapter will address the issues surrounding conducting general meta-analyses versus population and setting-specific meta-analyses in the career area.With meta-analysis, a researcher calculates an effect size for each study, typically by subtracting the control group mean from the treatment group mean and dividing by the pooled standard deviation of the groups. Therefore, if the treatment group mean is larger than the control group, the effect size is positive and reflects the degree or magnitude of the treatment group's effect. These effect sizes are weighted, on factors such as sample size, and are combined to produce an average effect size for the intervention studies. The content of the combined intervention studies is important in this process, along with whether it is better to more narrowly restrict studies to specific populations (e.g., AfricanAmericans) or settings (e.g., career development centers) or to more broadly include studies, which will be labeled general meta-analyses.Certainly there are advantages to having more specific meta-analyses. For example, a career counselor working with African American clients could search databases such as PSYCINFO for meta-analyses of career interventions with African Americans. A recent search of PSYCINFO with the keyword meta-analysis revealed over 16,000 studies. It would be difficult for the career counselor to review each of the studies in the hopes of identifying a meta-analysis pertinent to career counseling. Thereby, a clinician could simply enter the terms African American and career interventions and theoretically identify a meta-analysis, if such a metaanalysis existed, that could inform the clinician on the types of career interventions associated with larger effect sizes. The clinician could then adopt those interventions shown to be most effective with African American clients and provide evidence-based practice. Although such a meta-analysis does not currently exist, the desire for such information is evident as many researchers have called for an exploration of the efficacy of career interventions with ethnic and racial minorities (Fouad, & Kantamneni, 2008;Leong & Flores, 2015). Not only is there a need for specific information regarding the effectiveness of career intervention with racial and ethnic minority clients, but there may also be a need for specific

Improving the Design and Use of Meta Analyses in Evidence-Based Practice

“…The use of MUTOS at the problem-formulation stage connects well to the ideas described by Whiston (2017), who argues that the field of career-interventions would benefit from more "setting-specific" meta-analyses.…”

Summary of Key Elements in Using Theory and Research to Improve Evidence-Based Practice

“…First, Nadya Fouad (2017) examines the overall role of theory and how it can guide efforts to minimize the gap between science and practice. Next, chapters by Steve Brown (2017), Susan Whiston (2017), and Betsy Becker (2017) offer various perspectives on the state of the art in conducting meta-analyses on the outcomes of career development interventions. Finally, Scott Solberg (2017) offers a personal reflection on efforts to link vocational psychology scholarship to specific policy initiatives that require interdisciplinary collaborations.…”

“…Coding for specific study characteristics can allow for a more refined attention to testing possible moderators. Whiston (2017) concludes with a call for more systematic research on career interventions. Both general and specific meta-analyses are needed, with particular attention to moderators in order to examine specialized needs across various populations and settings.…”

“…Building on the knowledge from prominent meta-analyses of career outcomes concluding that career interventions are effective, there is a critical need to conduct more primary intervention research (Whiston, 2017), especially across diverse samples (Brown, 2017) involving a range of presenting concerns and settings. Becker (2017) featured the MUTOS method that enables researchers to examine sources of variability in outcomes across studies, including potential moderators involving client characteristics and features of interventions.…”