Evaluation Practices of Instructional Designers and Organizational Supports and Barriers

Abstract

The overall goal of this study was to determine the status quo of evaluation efforts of instructional designers as part of a systematic instructional system. This descriptive research investigated what organizational and managerial frameworks are required to sustain the evaluate component of systematic instructional design. Specifically, this survey: 1) Identifies and describes current evaluation practices and strategies of instructional designers within corporate and higher educational settings, 2) Describes types and causes of organizational supports and barriers instructional designers encounter as related to conducting instructional evaluation, and 3) Compares and contrasts professional practices and organizational issues by industry.

Training professionals' usage and understanding of Kirkpatrick's Level 3 and Level 4 evaluations

“…Based on the findings of this study, which correspond to the findings from Moller and Mallin (), Twitchell et al . (), Pulichino () and ASTD (), two of the primary reasons for this appear to be related to organizational support of evaluation activities.…”

“…Surprisingly, there have been only a small number of studies that investigated barriers to conducting behavior‐based Level 3 and results‐based Level 4 evaluations in organizations. Moller and Mallin () gathered qualitative data from their survey of instructional designers, asking them to discuss the presence of organizational barriers that hindered their efforts to conduct evaluations at any level. Eighty‐eight per cent of the respondents stated that such barriers existed in their organizations, with the dominant barrier being related to a lack of time and other resources to conduct evaluations.…”

“…Eighty‐eight per cent of the respondents stated that such barriers existed in their organizations, with the dominant barrier being related to a lack of time and other resources to conduct evaluations. Moller and Mallin () also documented responses related to organizational cultures resistant to evaluation, and lack of access to data required to conduct evaluations. They reported that only a small number of respondents identified a lack of ability or methodology as a barrier.…”

“…For example, post‐training interviews with instructors was considered by a respondent to be relevant data collection for a Level 3 evaluation. Moller and Mallin () noted this seemed to stem from a lack of knowledge about evaluation methodology, speculating that respondents do not truly understand what measurements indicate changes in behavior or how to take those measurements.…”

Understanding and representation of organizational training programs and their evaluation

“…achievement. Yet merely 20% of those surveyed correctly identified methods for results evaluation (Moller & Mallin, 1996;Wang and Wilcox, 2006). Brown & Gerhardt (2002) concluded that companies expend even less effort in evaluating the instructional design process (Nadler & Nadler, 2012;Wang & Wilcox, 2006).…”