This study investigates diffusion tensor imaging (DTI) for providing microstructural insight into changes in arterial tissue by exploring how cell, collagen and elastin content effect fractional anisotropy (FA), mean diffusivity (MD) and tractography. Five ex vivo porcine carotid artery models (n = 6 each) were compared—native, fixed native, collagen degraded, elastin degraded and decellularised. Vessels were imaged at 7 T using a DTI protocol with b = 0 and 800 s/mm2 and 10 isotopically distributed directions. FA and MD were evaluated in the vessel media and compared across models. FA values measured in native (p < 0.0001), fixed native (p < 0.0001) and collagen degraded (p = 0.0018, p = 0.0016, respectively) were significantly higher than those in elastin degraded and decellularised arteries. Native and fixed native had significantly lower MD values than elastin degraded (p < 0.0001) and decellularised tissue (p = 0.0032, p = 0.0003, respectively). Significantly lower MD was measured in collagen degraded compared with the elastin degraded model (p = 0.0001). Tractography yielded helically arranged tracts for native and collagen degraded vessels only. FA, MD and tractography were found to be highly sensitive to changes in the microstructural composition of arterial tissue, specifically pointing to cell, not collagen, content as the dominant source of the measured anisotropy in the vessel wall.