This paper reviews empirical research on the use of evaluation from 1986 to 2005 using Cousins and Leithwood’s 1986 framework for categorizing empirical studies of evaluation use conducted since that time. The literature review located 41 empirical studies of evaluation use conducted between 1986 and 2005 that met minimum quality standards. The Cousins and Leithwood framework allowed a comparison over time. After initially grouping these studies according to Cousins and Leithwood’s two categories and twelve characteristics, one additional category and one new characteristic were added to their framework. The new category is stakeholder involvement, and the new characteristic is evaluator competence (under the category of evaluation implementation). Findings point to the importance of stakeholder involvement in facilitating evaluation use and suggest that engagement, interaction, and communication between evaluation clients and evaluators is critical to the meaningful use of evaluations.

There is increasing interest in health care organizations functioning as learning health systems (LHSs) to improve the quality and efficiency of health care delivery while generating new knowledge. Individuals must be trained in associated concepts and competencies and subsequently positioned (or embedded) within the delivery system for maximum effect as they perform their scholarship. Potential researchers within LHSs come from many different training backgrounds; therefore, each LHS scholar requires a goal-directed plan tailored to his or her needs. There are few tools available to guide development, training, or evaluation of individuals interested in becoming leaders of research in LHSs. In this paper, we present a newly developed tool for guiding the training of such researchers, the Learning Health Systems Competency Appraisal Inventory (LHS-CAI). The LHS-CAI is modeled after the Clinical Research Appraisal Index (CRAI) used within Clinical and Translational Science Award sites across the United States. The LHS-CAI is a tool for trainees at all levels to use with their mentors in an interactive manner. The tool can then identify areas in which more training is needed and at what level to ensure success as a researcher within LHSs. We further modified the CRAI format to better leverage the LHS-CAI as a key part of an LHS scholar's individual development plan. To implement the LHS-CAI, we have identified key points within the Minnesota Learning Health System Mentored Career Development Program (MN-LHS) at which assessment of expertise for each competency would be useful to LHS scholars, mentors, and program leaders. Scholars in this program come from various clinical and academic backgrounds but are all targeting their career trajectories toward leading embedded LHS research. They will reevaluate their expertise upon completion of the program, with comparison to baseline serving as a key program evaluation tool. The LHS-CAI is currently being implemented with the first cohort of scholars in the MN-LHS program.

No abstract