The Validation of the Evaluation Involvement Scale for Use in Multisite Settings

Abstract

Evaluation researchers and practitioners acknowledge that involving stakeholders in the planning and implementation of an evaluation increases buy-in, understanding, and use. With the recent increase in multi-site evaluations of large federal programs, evaluators must think differently about how to encourage meaningful collaboration by stakeholders. To date, there has been no published measure of such involvement, despite recent calls for more systematic, replicable research. The purpose of this study was to validate the Evaluation Involvement Scale for use in multi-site evaluations. Between the fall of 2006 and spring of 2007 data were collected from an electronic survey and phone interviews of evaluators and principal investigators of four National Science Foundation program evaluations. Using Messick's unitary concept of validity as a framework, theoretical, statistical, and rational evidence is provided to support the use of the Evaluation Involvement Scale to measure stakeholder involvement in multi-site evaluations.

The relationship between involvement in and use of evaluation in multi-site evaluations

“…In previous studies, the interview data provided support for Toal's (2009) involvement in evaluation construct and Roseland's (2011) use/influence of evaluation construct (Table 7). For this study, the interview data was re-analyzed to provide insight into the relationship between the two constructs.…”

“…Previous studies have detailed the process of developing the constructs (see Roseland, 2011;Toal, 2009), while this study extends that work to examine the relationship between involvement in and use/influence of evaluation by performing a canonical correlation analysis with the constructs and further analysis of the interview data.…”

“…The survey was carefully designed to align with existing use/influence and involvement theory and was developed using think-alouds, pilot testing, and iterative revisions (see Toal, 2009). The entire population of principal investigators (PIs), co-principal investigators (co-PIs), and evaluators for each of the three NSF programs: ATE (limited opportunity for involvement), CETP (high opportunity for involvement), and LSC (middle level opportunity for involvement) were invited to participate in an online survey, resulting in a study population of 770 project PIs and evaluators.…”

Unexpected but Most Welcome

“…Respondents' opinions were measured on an ordinal scale of agreement and an open-ended question asked respondents to justify their choice (i.e., ''Why?''). The second part of the survey contained 11 four-point Likert-type questions from which the calculation of the Evaluation Involvement Scale (EIS) was derived (Toal, 2009). A total of 44 e-mail invitations were sent to the authors of the evaluation cases (the figure is more than 40 because second authors were contacted in cases of noncontacts and refusal to participate).…”

The role of involvement and use in multisite evaluations

“…Therefore, the study collected data from individual people serving as representatives of their projects. To highlight this distinction, we used the term involvement rather than participation (Lawrenz & Huffman, 2003;Toal, 2009;Toal, King, Johnson, & Lawrenz, 2009). Second, this research focused on the use of evaluation by secondary, somewhat unintended users.…”