Aging involves complex biological changes that affect disease susceptibility and aging trajectories. Although females typically live longer than males, they have a higher susceptibility to diseases like Alzheimer’s, speculated to be influenced by menopause, and reduced ovarian hormone production. Understanding sex-specific differences is crucial for personalized medical interventions and gender equality in health. Our study aims to elucidate sex differences in regional cerebellar structure and connectivity during normal aging by investigating both structural and functional connectivity variations, with a focus on investigating these differences in the context of sex-steroid hormones. The study included 138 participants (mean age = 57(13.3) years, age range = 35-86 years, 54% women). The cohort was divided into three groups: 38 early middle-aged individuals (EMA) (mean age = 41(4.7) years), 48 late middle-aged individuals (LMA) (mean age = 58(4) years), and 42 older adults (OA) (mean age = 72(6.3) years). All participants underwent MRI scans, and saliva samples were collected for sex-steroid hormone quantification (17β-estradiol (E), progesterone (P), and testosterone (T)). We found less connectivity in females between Lobule I-IV and the cuneus, and greater connectivity in females between Crus I, Crus II, and the precuneus with increased age. Higher 17β-estradiol levels were linked to greater connectivity in Crus I and Crus II cerebellar subregions. Analyzing all participants together, testosterone was associated with both higher and lower connectivity in Lobule I-IV and Crus I, respectively, while higher progesterone levels were linked to lower connectivity in females. Structural differences were observed, with EMA males having larger volumes compared to LMA and OA groups, particularly in the right I-IV, right Crus I, right V, and right VI. EMA females showed higher volumes in the right lobules V and VI. These results highlight the significant role of sex hormones in modulating cerebellar connectivity and structure across adulthood, emphasizing the need to consider sex and hormonal status in neuroimaging studies to better understand age-related cognitive decline and neurological disorders.