What makes a good quality indicator set? A systematic review of criteria

Abstract

Background While single indicators measure a specific aspect of quality (e.g. timely support during labour), users of these indicators such as patients, providers and policy-makers are typically interested in some broader construct (e.g. quality of maternity care) whose measurement requires a set of indicators. However, guidance on desirable properties of indicator sets is lacking. Based on the premise that a set of valid indicators does not guarantee a valid set of indicators, the aim of this review is twofold: First, we introduce content validity as a desirable property of indicator sets and review the extent to which studies in the peer-reviewed health care quality literature address this criterion. Second, to obtain a complete inventory of criteria, we examine what additional criteria of quality indicator sets were used so far. Methods : We searched the databases Web of Science, Medline, Cinahl and PsycInfo from inception to May 2021 and the reference lists of included studies. English- or German-language, peer-reviewed studies concerned with desirable characteristics of quality indicator sets were included. Applying qualitative content analysis, two authors independently coded the articles using a structured coding scheme and discussed conflicting codes until consensus was reached. Results Of 366 studies screened, 62 were included in the review. 85% (53/62) of studies addressed at least one of the component criteria of content validity (content coverage, proportional representation, contamination) and 15% (9/62) addressed all component criteria. Studies used various content domains to structure the targeted construct (e.g., quality dimensions, elements of the care pathway, policy priorities), providing a framework to assess content validity. The review revealed four additional substantive criteria for indicator sets: cost of measurement (21% [13/62] of the included studies), prioritization of “essential” indicators (21% [13/62]), avoidance of redundancy (13% [8/62]) and size of the set (15% [9/62]). Additionally, four procedural criteria were identified: stakeholder involvement (69% [43/62]), using a conceptual framework (44% [27/62]), defining the purpose of measurement (26% [16/62]) and transparency of the development process (8% [5/62]). Conclusion The concept of content validity and its component criteria help assessing whether conclusions based on a set of indicators are valid conclusions about the targeted construct. To develop a valid indicator set, careful definition of the targeted construct including its (sub-)domains is paramount. Developers of quality indicators should specify the purpose of measurement and consider trade-offs with other criteria for indicator sets whose application may reduce content validity (e.g. costs of measurement) in light thereof.

A multi-stage process to develop quality indicators for community-based palliative care using interRAI data

“…While it is important to understand the validity of individual indicators, it is also important to evaluate the content validity of the set of QIs. Several criteria have been proposed with which to carry out this evaluation [ 62 ]. For example, it is important that the QIs adequately cover the depth and breadth of the content of interest.…”

Assessing Resource Efficiency of City Neighbourhoods: A Methodological Framework for Structuring and Practical Application of Indicators in Urban Planning

“…Many publications address the compilation of groups of indicators, but few tackle the question concerning how to evaluate the quality of an indicator set in general. Motivated by the field of healthcare, but in a very generic way, Schang et al [28] discuss the question of how to describe a good-quality indicator set. Starting from the premise "that a set of 'valid indicators' does not guarantee a 'valid set' of indicators", they propose that content validity should be constitutive for indicator sets, in the sense of ensuring that the content of the assessment instrument adequately reflects the targeted construct [28].…”

Should indicators be correlated? Formative indicators for healthcare quality measurement

“…With formative indicators, indicator selection should be guided by the criterion of content validity, which is defined as the degree to which indicators represent all relevant aspects of the targeted quality topic properly 13 20. A content-valid indicator set enables users of measurement results such as patients, providers and policy makers to draw appropriate conclusions about a provider’s quality with regard to the measured quality topic.…”

“…A content-valid indicator set enables users of measurement results such as patients, providers and policy makers to draw appropriate conclusions about a provider’s quality with regard to the measured quality topic. To assure content validity of an indicator set, it is important to clearly specify the relevant content domains and to select indicators commensurate with these domains by involving stakeholders 20…”