Critical Appraisal of the Design and Reporting of Studies of Imaging and Measurement of Carotid Stenosis

Abstract

Background and Purpose-Several hundred studies have been published over the last few years on imaging and measurement of carotid stenosis. Despite all this research, there is still no consensus about how best to image and measure stenosis. One possible explanation for this is that many of the studies have not been large enough or methodologically sound enough to allow useful conclusions to be drawn. We aimed to assess the design and methods of a random sample of published studies of imaging and measurement of carotid stenosis using 9 simple criteria. Methods-A formal literature search was performed for studies of imaging and measurement of carotid stenosis. Two subsets were randomly selected for detailed assessment: 20 studies published before 1991 and 20 published between 1993 and 1997 (some years after the initial publication of the ECST and NASCET trials). The criteria used to assess the selected studies were as follows: prospective rather than retrospective study design; patient selection based on a consecutive series or a random sample; adequate detail of study population; adequate detail of imaging techniques; inclusion of all investigations, ie, patients with poor-quality imaging were not excluded; blinded assessment of images; adequate detail of derivation of measurement of stenosis from images or data; adequate data on the reproducibility of measurements of stenosis; and study powered according to a sample-size calculation. Results-There were many basic methodological deficiencies in both subsets of studies, with relatively little evidence of improvement with time. For example, only 33% of studies were prospective, only 45% studied a consecutive or random selection of patients, and only 38% reported any data on the reproducibility of measurements. More than half of the studies satisfied Յ4 of the 9 quality criteria. However, there was considerable variation between studies, with 7 studies satisfying Ն7 criteria and 10 studies satisfying Յ2. No study was based on a sample-size calculation. The number of patients studied was often small, particularly in the more recent studies: median sample size was 100 in the 1970 -1990 studies and 58 in the 1993-1997 studies (PϽ0.0001). Conclusions-The design and reporting of published studies of imaging and measurement of carotid stenosis are poor and have not improved much in recent years. The majority of published studies are not of a sufficient standard to enable the results to be used to inform clinical practice. The utility of future studies could be improved considerably by better adherence to 9 simple methodological guidelines.

Accurate, practical and cost-effective assessment of carotid stenosis in the UK

“…[28][29][30] A recent literature search identified over 2500 primary papers on MRA alone. 31 Small, very positive studies, with detailed descriptions of the imaging technique but woefully inadequate information on the patients or proportion of severe stenoses, which omit to mention whether the imaging assessment was blinded to the reference standard, rarely give details of how many images were excluded because they were 'suboptimal', give no information on observer reliability and fail to seek patients' opinions, are all too common.…”

“…31 Small, very positive studies, with detailed descriptions of the imaging technique but woefully inadequate information on the patients or proportion of severe stenoses, which omit to mention whether the imaging assessment was blinded to the reference standard, rarely give details of how many images were excluded because they were 'suboptimal', give no information on observer reliability and fail to seek patients' opinions, are all too common. 28,29,32 Some of this is due to genuine difficulty in conducting such studies (e.g. sample size), but some is just lack of awareness of good methodology.…”

“…Instead, there was often overemphasis on technical details of the imaging technique used; these are also important, but not to the exclusion or omission of basic study design factors. 29,39 It is hoped that initiatives such as the STARD criteria (Appendix 3 and http://www.consortstatement.org/stardstatement.htm) and the Cochrane Screening and Diagnostic Test will help to focus attention on the importance of design and reporting of primary studies of diagnostic test accuracy.…”

“…29 Both found methodological details lacking and major study design flaws in the primary imaging literature. Without wishing to be overcritical, it would seem that these seven previous systematic reviews have not recognised several key methodological points in the assessment of less invasive imaging tests.…”

“…28,29 Moreover, where studies were conducted to an acceptable standard, the results may not have been presented in the detail required to answer the question posed by the systematic reviewer. For example, in the systematic review described in Chapter 3, seven studies 81,92,95,114,117,119,123 did not give data in the diagnostic categories of 0-49 or occluded, 50-69% and 70-99% stenosis by the NASCET criteria or equivalent.…”