Background and Objectives: Positron emission tomography (PET) with amyloid-ligands recently gained interest for evaluating white matter (WM) pathology. Here, we investigated the neurobiological underpinnings of the amyloid-PET signal in the WM using free water (FW) diffusion MRI in a mixed cohort of small vessel disease (SVD) and Alzheimer's disease (AD) pathology. Methods: We included data from two large cohort imaging studies, including MITNEC-C6 with moderate-to-severe white matter hyperintensity (WMH) burden and ADNI-2 with mild-to-moderate WMH burden, covering the spectrum of cognitively normal to dementia. Subjects underwent diffusion MRI, 18F-AV45 amyloid-PET, and cognitive testing. We calculated WM diffusion metrics including fractional anisotropy (FA) and mean diffusivity (MD). In addition, we applied a bi-tensor diffusion MRI model that differentiates between extracellular (FW fraction) and tissue-specific (FW-adjusted FA) compartments of the WM. We tested associations of all diffusion metrics with 18F-AV45 SUVR in regions of WMH vs. normal-appearing WM (NAWM), and with cognition. We further performed partial-least-square analysis to investigate how the diffusion metrics as well as age, sex, education, WMH volume, and cortical 18F-AV45 SUVR covary with WM 18F-AV45 SUVR. Results: 18F-AV45 SUVR was significantly lower in regions of WMH compared to NAWM (t=25.08, P<0.0001, n=115 subjects). Within WMH, lower 18F-AV45 SUVR was associated with higher FW (β=-0.36±0.13, P=0.005) and lower FA (β=+0.24±0.12, P=0.046), but not with the tissue-specific metric FW-adjusted FA. Partial-least-square analysis further confirmed that FW had the most influence on 18F-AV45 SUVR in regions of WMH. In contrast, FW-adjusted FA had more influence on 18F-AV45 SUVR in NAWM. Last, correlation with cognitive impairment was higher for FW than FW-adjusted FA, both in regions of WMH (MMSE: βFW=-0.40±0.13, P=0.003; βFW-adjustedFA=0.14±0.09, P=0.11) and NAWM (MMSE: βFW=-0.30±0.11, P=0.01; βFW-adjustedFA=0.21±0.09, P=0.02. Conclusion: In a cohort representative of the more common AD population, reduced amyloid-PET uptake in WM lesions may largely reflect the appearance of extracellular FW, while changes in NAWM may associate with tissue-specific damage. Ultimately, our findings may support FW and amyloid-PET as markers of WM abnormalities in SVD and AD.