Sleep is a ubiquitous phenomenon, essential to the organism homeostasis. Notwithstanding, there has been an increasing concern with its disruption, not only within the context of pathological conditions, such as neurologic and psychiatric diseases, but also in health. In fact, sleep complaints are becoming particularly common, especially in middle-aged and older adults, which may suggest an underlying susceptibility to sleep quality loss and/or its consequences. Thus, a whole-brain modeling approach to study the shifts in the system can cast broader light on sleep quality mechanisms and its associated morbidities. Following this line, we sought to determine the association between the standard self-reported measure of sleep quality, the Pittsburgh Sleep Quality Index (PSQI) and brain correlates in a normative aging cohort. To this purpose, 86 participants (age range 52–87 years) provided information regarding sociodemographic parameters, subjective sleep quality and associated psychological variables. A multimodal magnetic resonance imaging (MRI) approach was used, with whole-brain functional and structural connectomes being derived from resting-state functional connectivity (FC) and probabilistic white matter tractography (structural connectivity, SC). Brain regional volumes and white matter properties associations were also explored. Results show that poor sleep quality was associated with a decrease in FC and SC of distinct networks, overlapping in right superior temporal pole, left middle temporal and left inferior occipital regions. Age displayed important associations with volumetric changes in the cerebellum cortex and white matter, thalamus, hippocampus, right putamen, left supramarginal and left lingual regions. Overall, results suggest that not only the PSQI global score may act as a proxy of changes in FC/SC in middle-aged and older individuals, but also that the age-related regional volumetric changes may be associated to an adjustment of brain connectivity. These findings may also represent a step further in the comprehension of the role of sleep disturbance in disease, since the networks found share regions that have been shown to be affected in pathologies, such as depression and Alzheimer's disease.