Prior diffusion tensor imaging (DTI) studies have investigated white matter (WM) changes in patients with primary restless legs syndrome (RLS), but the results were inconsistent. Here, we proposed using tract-specific statistical analysis (TSSA) to find alterations in specific WM tracts to clarify the pathophysiological mechanisms of RLS. We enrolled 30 patients with RLS and 31 age- and sex- matched controls who underwent brain magnetic resonance imaging, neuropsychological tests, and polysomnography. Fractional anisotropy (FA) maps obtained from whole-brain diffusion tensor imaging and TSSA were used to localize WM changes in patients with RLS. Subsequently, a comparison of FA values for each tract between patients and controls was performed. The associations between FA values and clinical, polysomnographic, and neuropsychological parameters in RLS patients were assessed. RLS patients demonstrated decreased FA values in the left corticospinal tract (CST) and cingulum, and in the right anterior thalamic radiation (ATR) and inferior fronto-occipital fasciculus (IFO). Patients’ attention/executive function and visual memory scores positively correlated with FA values in the right ATR, and anxiety levels negatively correlated with FA values in the right IFO. Additionally, the number of periodic leg movements and movement arousal index were negatively correlated with FA values in the left CST. The TSSA method identified previously unknown tract-specific alterations in patients with RLS and significant associations with distinct clinical manifestations of RLS.