Understanding Dimensions of Organizational Evaluation Capacity

Abstract

Organizational evaluation capacity building has been a topic of increasing interest in recent years. However, the actual dimensions of evaluation capacity have not been clearly articulated through empirical research. This study sought to address this gap by identifying the key dimensions of evaluation capacity in Canadian federal government organizations. The methodology used, based on Leithwood and Montgomery’s Innovation Profile approach, featured semistructured interviews with evaluation experts and a validating exercise conducted in four government organizations. The framework developed as a result of the study identifies six main dimensions of evaluation capacity (human resources, organizational resources, evaluation planning and activities, evaluation literacy, organizational decision making, and learning benefits), each one broken down into further subdimensions. The evaluation capacity of organizations on each of these dimensions and subdimensions can be described using four levels: low, developing, intermediate, and exemplary. The study found that government organizations vary in terms of their capacity from one dimension to the next, and indeed, from one subdimension to the next.

Strategic Evaluation Utilization in the Canadian Federal Government

“…Second, the authors cite tacit resistance to evaluation on the part of program managers. More recent research on organizational evaluation capacity building confirms that organization-wide evaluation literacy tends to promote evaluation utilization (e.g., see Bourgeois & Cousins, 2013;Newcomer, 2007). Third, and most interestingly in our current context, ".…”

“…The concept of EC, therefore, constitutes an important element of our study's theoretical base. Even though several theoretical and practical frameworks have been developed to describe the various components of organizational evaluation capacity, we elected to apply our own framework (Bourgeois & Cousins, 2013) as the conceptual backdrop to our work; this framework was developed through an empirical study conducted with four federal government organizations and thus reflects the language, practices, and structures for evaluation found in these organizations, while other models of EC apply to other types of organizations. The framework, shown in its entirety by Bourgeois and Cousins (2013), essentially identifies six dimensions of EC: three of these dimensions illustrate an organization's capacity to do evaluation (Human Resources, Organizational Resources, and Evaluation Planning and Activities), while the other three dimensions focus on an organization's capacity to use evaluation (Evaluation Literacy, Organizational Decision-Making, and Learning Benefits.…”

“…Even though several theoretical and practical frameworks have been developed to describe the various components of organizational evaluation capacity, we elected to apply our own framework (Bourgeois & Cousins, 2013) as the conceptual backdrop to our work; this framework was developed through an empirical study conducted with four federal government organizations and thus reflects the language, practices, and structures for evaluation found in these organizations, while other models of EC apply to other types of organizations. The framework, shown in its entirety by Bourgeois and Cousins (2013), essentially identifies six dimensions of EC: three of these dimensions illustrate an organization's capacity to do evaluation (Human Resources, Organizational Resources, and Evaluation Planning and Activities), while the other three dimensions focus on an organization's capacity to use evaluation (Evaluation Literacy, Organizational Decision-Making, and Learning Benefits. The three dimensions that are of particular conceptual interest for the present study are those related to organizational capacity to use evaluation: 1) Evaluation Literacy refers to the broad knowledge of evaluation across the organization; 2) Organizational Decision-Making focuses on the instrumental and conceptual application of evaluation findings in broader processes, such as program development or budget reallocation exercises; and 3) Learning Benefits focuses on applying knowledge derived from evaluations into everyday activities and organizational learning (including instrumental and process uses).…”

“…Given that organizational evaluation capacity is known to be strongly connected to all types of evaluation use (e.g., Bourgeois & Cousins, 2013), respondents were asked to comment on the overall knowledge of program staff and managers regarding evaluation. In general, respondents in both organizations felt that organizational knowledge about evaluation is high, particularly among program managers.…”

Higher education reforms in Romania. Between the Bologna process and national challenges

“…Evaluation knowledge and practice become better understood and increasingly used in organizations which resort to the implementation of intentional ECB strategies (Bourgeois and Cousins 2013). Shaping evaluation priorities at the organizational level implies developing processes like: (a) identifying important topics for dealing with organization's mission and objectives; (b) analysing the topics and revealing the logical connections between them; and (3) arrange them by previously established criteria and select priorities.…”