Advisory groups for evaluations in diverse cultural groups, communities, and contexts

Abstract

An advisory group constructed from within the community where a program evaluation is conducted can be an invaluable resource to an evaluator, particularly if the evaluator is an outsider and of a different culture. The author identifies useful roles that advisory groups have played in his organization's evaluations, and explores advisory group selection criteria, processes for identifying, vetting, and recruiting potential members who meet these criteria, and recruitment-related pitfalls. Shortcomings of advisory groups in these contexts are discussed. The authors examine dos, don'ts, and lessons learned from working effectively with advisory groups in diverse cultural contexts; for example, understanding and appreciating unfamiliar styles of discourse and patterns of interaction and how to adapt to these. Differences in age, gender, position, social status, education, literacy, national origin, dialect, degree of assimilation, political affiliation, clan membership, knowledge of English, and comfort with European-American manners and customs are all at play.

Evaluation Advisory Groups: Considerations for Design and Management

“…• improve evaluation appropriateness-for example, attending to ethical, cultural, and political sensitivities CDC, 2011 ;Johnston-Goodstar, 2012); • enhance technical/methodological quality-particularly where external methodological expertise would add value CDC, 2011 ); • validate evaluation design, tools, processes, and results ( CDC, 2011 ;Fetterman et al, 2018;Johnston-Goodstar, 2012); • facilitate navigation and logistics-help to open otherwise closed doors, point to easier ways ( Cohen, 2012 ;VeLure Roholt & Baizerman, 2012a); • facilitate data access/collection-particularly with complex interventions, dynamic contexts, and special populations (e.g., children, vulnerable, etc.) ( CDC, 2011 ;Cohen, 2012 ;VeLure Roholt, 2012 ); • champion and represent-act as ambassadors to improve interactions and overcome tensions among diverse stakeholders and/or sectors (Fetterman et al, 2018); • provide a way for individuals to speak to their organizations-the group can speak to issues which individuals do not have permission to speak (Fetterman et al, 2018, Johnston-Goodstar, 2012); • generate conclusions and recommendations-ones that are appropriate, helpful, and/or actionable based on members' understanding of their organizations and the larger context ( CDC, 2011 ;Fetterman et al, 2018;Johnston-Goodstar, 2012); • facilitate utilization-by personally representing implications and recommendations as insiders, and helping to overcome internal and interorganizational barriers to decisions for positive change Johnston-Goodstar, 2012); and • facilitate evaluation capacity building-in general as well as in specifi c skill areas and/or in knowledge mobilization and utilization ( CDC, 2011 ;Cohen, 2012 ;Fetterman et al, 2018).…”

“…Th ere is no consensus in the literature on the right number of people to include in an EAG. Cohen (2012 ) recommends fi ve to seven members; Richards-Schuster (2012 ) created a youth advisory group with 12 members; CDC (2011 ) created an EAG with 32 members. Small EAGs, coupled with good facilitation, can help ensure that everyone has a chance to contribute to the dialogue.…”

“…Recruitment is a critical process, but can be time-consuming ( Cohen, 2012 ). Options to identify EAG members include gathering recommendations from partners, community directories, and newspaper articles related to the topic, program, or organization.…”

“…Options to identify EAG members include gathering recommendations from partners, community directories, and newspaper articles related to the topic, program, or organization. When recruiting EAG members, the evaluator must be aware of diverse perspectives so these are represented ( Cohen, 2012 ).Th e evaluator can ask EAG members to identify who else needs to be at the table. Th ere may be a need for the evaluator and client to screen potential members for fi t, and to prepare them so they can contribute eff ectively (Baizerman et al, 2014 ;VeLure Roholt & Baizerman, 2012a, 2012b.…”

Building Evaluation Capacity in Youth-Serving Organizations Through Evaluation Advisory Boards

“…Engaging board champions early in the process of creating an EAB helps facilitate recruitment because board member candidates are more likely to join a body with enthusiastic colleagues who appeal to their work interests. These early champions can also be helpful in establishing a recruitment approach to identify qualified candidates recommended by fellow members (Cohen, 2012), while also being intentional about engaging diverse members who bring a variety of perspectives. In doing so, youth-serving organizations can be mindful of how their research and evaluation environments (including, for example, the creation of an EAB) offer equitable opportunities (Outley & Blyth, 2020).…”