There is great value to use of structural neuroimaging in the assessment of Alzheimer's disease (AD). However, to date, predictive value of structural imaging tend to range between 80% and 90% in accuracy and it is unclear why this is the case given that structural imaging should parallel the pathologic processes of AD. There is a possibility that clinical misdiagnosis relative to the gold standard pathologic diagnosis and/or additional brain pathologies are confounding factors contributing to reduced structural imaging classification accuracy. We examined potential factors contributing to misclassification of individuals with clinically diagnosed AD purely from cortical thickness measures. Correctly classified and incorrectly classified groups were compared across a range of demographic, biological, and neuropsychological data including cerebrospinal fluid biomarkers, amyloid imaging, white matter hyperintensity (WMH) volume, cognitive, and genetic factors. Individual subject analyses suggested that at least a portion of the control individuals misclassified as AD from structural imaging additionally harbor substantial AD biomarker pathology and risk, yet are relatively resistant to cognitive symptoms, likely due to "cognitive reserve," and therefore clinically unimpaired. In contrast, certain clinical control individuals misclassified as AD from cortical thickness had increased WMH volume relative to other controls in the sample, suggesting that vascular conditions may contribute to classification accuracy from cortical thickness measures. These results provide examples of factors that contribute to the accuracy of structural imaging in predicting a clinical diagnosis of AD, and provide important information about considerations for future work aimed at optimizing structural based diagnostic classifiers for AD.