Potholes and Molehills: Bias in the Diagnostic Performance of Diffusion-Tensor Imaging in Concussion

Abstract

Purpose:To investigate the extent of bias in a clinical study involving "pothole analysis" of diffusion-tensor imaging (DTI) data used to quantify white matter lesion load in diseases with a heterogeneous spatial distribution of pathologic findings, such as mild traumatic brain injury (TBI), and create a mathematical model of the bias. Materials and Methods:Use of the same reference population to define normal findings and make comparisons with a patient group introduces bias, which potentially inflates reported diagnostic performance. In this institutional review board-approved prospective observational cohort study, DTI data were obtained in 20 patients admitted to the emergency department with mild TBI and in 16 control subjects. Potholes and molehills were defined as clusters of voxels with fractional anisotropy values more than 2 standard deviations below and above the mean of the corresponding voxels in the reference population, respectively. The number and volume of potholes and molehills in the two groups were compared by using a Mann-Whitney U test. Results:Standard analysis showed significantly more potholes in mild TBI than in the control group (102.5 6 34.3 vs 50.6 6 28.9, P , .001). Repeat analysis by using leave-one-out cross-validation decreased the apparent difference in potholes between groups (mild TBI group, 102.5 6 34.3; control group, 93.4 6 27.2; P = .369). It was demonstrated that even with 100 subjects, this bias can decrease the voxelwise false-positive rate by more than 30% in the control group. Conclusion:The pothole approach to neuroimaging data may introduce bias, which can be minimized by independent training and test groups or cross-validation methods. This bias is sufficient to call into question the previously reported diagnostic performance of DTI for mild TBI.q RSNA, 2014

Traumatic Brain Injury as a Disorder of Brain Connectivity

“…Thus, the bulk of studies examining military blast mTBI have included individuals with both primary blast exposure and additional secondary and tertiary injuries. These studies have generally shown decreases in white matter integrity both in the subacute (Mac Donald et al, 2011) and chronic phase (Davenport et al, 2012; Hayes et al, 2015; Jorge et al, 2012; Mac Donald et al, 2011; Morey et al, 2013), similar to the civilian mTBI literature (but see Bazarian et al, 2013; Levin et al, 2010; Watts et al, 2014). However, findings of greater neurologic deficits in individuals exposed to combat-related mTBI than civilian mTBI (Ruff et al, 2012) provide some evidence for increased noxious effects due to blast and suggest the need for further studies comparing blast-induced mTBI and civilian TBI.…”

White matter abnormalities in mild traumatic brain injury with and without post-traumatic stress disorder: a subject-specific diffusion tensor imaging study

“…To summarize: clusters of voxels with abnormal values (e.g., z-scores beyond a given threshold) are calculated at the subject-specific analysis level before they are compared between groups. Although this approach has shown promise in adult and pediatric DTI studies of mTBI (Jorge et al 2012; Ling et al 2012; Mayer et al 2012; Miller et al 2016), the absence of cross-validation methods (e.g., leave-one-out; see Methods) in those studies raises the risk of bias (Mayer et al 2014; Watts et al 2014). …”

Comparative Neuropsychiatry: White Matter Abnormalities in Children and Adolescents with Schizophrenia, Bipolar Affective Disorder, and Obsessive-Compulsive Disorder

“…In addition, the size of the potholes was considerably larger than those in the EOS group. Second, concern has surface recently over potential bias that can be associated with the pothole approach when the sample size for the control group is small [70]. While this bias clearly applies for small cluster sizes, it is not necessarily true for larger cluster sizes of contiguous voxels.…”

“…Interestingly, the children and adolescents with OCD and the controls had nearly a perfect overlap, which would not be expected in the presence of bias unless the group of children and adolescents with OCD has, on average, less WM potholes than the controls. When bias correction was applied as suggested by Watts et al [70], the EOB group had a significantly greater number and volume of potholes, whereas the OCD group had significantly fewer potholes compared to controls.…”