Frameworks for evaluating education programmes and work related learning: a scoping review

Abstract

Purpose It has become accepted practice to include an evaluation alongside learning programmes that take place at work, as a means of judging their effectiveness. There is a tendency to focus such evaluations on the relevance of the intervention and the amount of learning achieved by the individual. The aim of this review is to examine existing evaluation frameworks that have been used to evaluate education interventions and, in particular, assess how these have been used and the outcomes of such activity. Design/methodology/approach A scoping review using Arskey and O’Malley’s five stage framework was undertaken to examine existing evaluation frameworks claiming to evaluate education interventions. Findings Forty five articles were included in the review. A majority of papers concentrate on learner satisfaction and/or learning achieved. Rarely is a structured framework mentioned, or detail of the approach to analysis cited. Typically, evaluations lacked baseline data, control groups, longitudinal observations and contextual awareness. Practical implications This review has implications for those involved in designing and evaluating work-related education programmes, as it identifies areas where evaluations need to be strengthened and recommends how existing frameworks can be combined to improve how evaluations are conducted. Originality/value This scoping review is novel in its assessment and critique of evaluation frameworks employed to evaluate work-related education programmes.

Evaluating a quality improvement collaborative: a hybrid approach

“…The evaluation model that has external recognition and dominance is that attributed to Kirkpatrick and Kirkpatrick (2006) and it could be argued that this 4 level framework (see Figure 1) is a necessary component of a thorough evaluation although not of itself sufficient (Caley et al. , 2021).…”

“…In considering the challenges of evaluation of a complex initiative we reviewed the circumstances and the work that had been initiated by the QIC project team before deciding which of the several models available might be of greatest use. We undertook a review of the literature on evaluation techniques in order to establish if, and how many, evaluations were conducted using an established framework (Caley et al, 2021). Our conclusion was that Kirkpatrick and Kirkpatrick (2006), whilst universally acclaimed, fell short of providing an adequate procedure for comprehensive judgements to be made.…”

“…The evaluation model that has external recognition and dominance is that attributed to Kirkpatrick and Kirkpatrick (2006) and it could be argued that this 4 level framework (see Figure 1) is a necessary component of a thorough evaluation although not of itself sufficient (Caley et al, 2021). The model is intuitively understandable and well regarded in practice for that reason, but it is sketchy on process and thus problematic in use.…”

Evaluation of a family liaison officer role introduced during the COVID‐19 pandemic: A mixed methods study

“…We used Stake's 10 framework (Figure 1) to assess the rationale, context and where possible the outcomes associated with the FLO role. This framework was selected as it is one of the few models that provides a comprehensive and multidimensional approach to examining the development, delivery and outcomes of an intervention, capturing all elements, all relevant data, and providing a clear strategy for analysis of findings, thus offering a comparable and credible evaluation pathway 11 …”

“…This framework was selected as it is one of the few models that provides a comprehensive and multidimensional approach to examining the development, delivery and outcomes of an intervention, capturing all elements, all relevant data, and providing a clear strategy for analysis of findings, thus offering a comparable and credible evaluation pathway. 11 We had two distinct but complementary foci. First, to establish the extent to which the role has been successful in meeting its objective to facilitate communication between patients and their relatives, while hospital visiting was heavily restricted.…”