The aim is to investigate the relationship between microstructural white matter (WM) diffusivity indices and macrostructural WM volume (WMV) among healthy individuals (20–85 years). Whole-brain diffusion measures were calculated from diffusion tensor imaging using FMRIB software library while WMV was estimated through voxel-based morphometry, and voxel-based analysis was carried out using tract-based spatial statistics. Our results revealed that mean diffusivity, axial diffusivity, and radial diffusivity had shown good correlation with WMV but not for fractional anisotropy (FA). Voxel-wise tract-based spatial statistics analysis for FA showed a significant decrease in four regions for middle-aged group compared to young-aged group, in 22 regions for old-aged group compared to middle-aged group, and in 26 regions for old-aged group compared to young-aged group (P < 0.05). We found significantly lower WMV, FA, and mean diffusivity values in females than males and inverted-U trend for FA in males. We conclude differential age- and gender-related changes for structural WMV and WM diffusion indices.