Setting and maintaining standards for patient-reported outcome measures: can we rely on the COSMIN checklists?

Abstract

As test-developers we have often been troubled by published reviews of patient-reported outcome measures (PROMs). Too often minor issues are judged important while other reviews exclude the best measures available. Perhaps this led several groups to make recommendations for evaluating the quality of PROMs. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist is the latest set of recommendations. While reviewing the COSMIN literature and reviews conducted using their recommendations several concerns became apparent. The checklist is not evidence-based, relying on the opinion of researchers experienced in health-related quality of life. PROMs measuring other types of outcomes are inadequately covered by the checklist. COSMIN choose to focus on Classical Test Theory and the checklists are not appropriate for use with PROMs developed using modern measurement. Such an approach only obstructs progress in the field of outcome measurement. The retrospective nature of the evaluations also penalizes new PROMs. While the checklists imply that composite, ordinal level measurement is acceptable, crucial aspects of instrument development and quality are excluded. Reviews based on the COSMIN checklist produce contradictory conclusions and fail to provide evidence to support the recommendations. These problems suggest that the checklists themselves lack reliability and validity. It is also clear that several reviewers lack the expertise to apply the checklists. Researchers require a good grounding in instrument development and psychometrics to produce quality reviews. The science of modern PROM development is still in an early phase. Few available PROMs have sufficient quality, limiting the need for complex reviews. Standards need to be agreed for high quality outcome measurement. The issue is who should set these standards? Most published reviews merely scratch the surface and lack essential detail. All reviews of PROMs should be treated with caution, irrespective of whether the COSMIN checklist was employed.

Reply to the concerns raised by McKenna and Heaney about COSMIN

“…We would like to respond to the paper by McKenna and Heaney 1 , in which they express their concerns with regard to the COSMIN methodology 2 . We appreciate the attention to our work, and more in general, to the quality of outcome measurement instruments.…”

“…McKenna and Heaney point to the importance of using conceptual models when developing measurement instruments and evaluating their measurement properties, and suggest that the COSMIN methodology doesn't pay enough attention to defining the construct by grounding it in a theory 1 . However, in the COSMIN Risk of Bias checklist, the standards 1 and 2 in Box 1 PROM development, refer to the clarity of the construct, and the use of a conceptual model, respectively 5 .…”

“…A second point these authors raised refers to the use of unidimensional scales, suggesting that within the COSMIN methodology the importance of this is ignored 1 . However, COSMIN also addresses that evidence is required on the unidimensionality of (sub)scales based on a reflective model.…”

“…McKenna and Heaney clearly explain the advantages of developing PROMs using Rasch or IRT analyses 1 . Indeed, the CLIQ PROM 10 developed by McKenna and colleagues, was developed based on qualitative studies, in combination with Rasch analyses.…”

“…A legitimate concern McKenna and Heaney raise is the difference in conclusions drawn by different authors of systematic reviews using the COSMIN methodology 1 . The question is whether this is due to the COSMIN methodology or to the expertise and use of COSMIN by these authors.…”