Most previous studies have explored the relationship between gray matter volume and sleep quality, but little is known about the relationship between white matter volume and sleep quality. Data were collected using the Pittsburgh Sleep Quality Index and voxel-based morphometry among 352 healthy college students. Results showed that the global PSQI score was negatively associated with the white matter volume, including in the right middle occipital gyrus, the left superior temporal gyrus, the right the precentral gyrus, the left supramarginal gyrus, the left middle frontal gyrus, the left precunes, and the right superior frontal gyrus. Results also indicated that the white matter volume in specific regions negatively predicted the factor of PSQI. These specific brain regions may be replicated in brain areas related to sleep quality. In summary, we suggested that an investigation of white matter structural alterations in the specific regions might be beneficial to tackle underlying neurological mechanisms of sleep quality.