(1) Background: Alzheimer’s disease (AD), an age-progressive neurodegenerative disease that affects cognitive function, causes changes in the functional connectivity of the default-mode network (DMN). However, the question of whether AD-related changes occur in the functional connectivity of the basal ganglia has rarely been specifically analyzed. This study aimed to measure the changes in basal ganglia functional connectivity among patients with AD and mild cognitive impairment (MCI) in their resting state using the functional connectivity density (FCD) value, the functional connectivity (FC) intensity, and the graph theory index, and to confirm their influence on clinical manifestations. (2) Methods: Resting-state functional MRI (rs-fMRI) and neuropsychological data from 48 participants in the Alzheimer’s Disease Neuroimaging Initiative (ADNI) were used for analyses. The 48 ADNI participants comprised 16 patients with AD, 16 patients with MCI, and 16 normal controls (NCs). The functional connectivity of basal ganglia was evaluated by FCDs, FC strength, and graph theory index. We compared voxel-based FCD values between groups to show specific regions with significant variation and significant connectivity from ROI conduction to ROI analysis. Pearson’s correlation analyses between functional connectivity and several simultaneous clinical variables were also conducted. Additionally, receiver operating characteristic (ROC) analyses associated with classification were conducted for both FCD values and graph theory indices. (3) Results: The level of FCD in patients with cognitive impairment showed obvious abnormalities (including short-range and long-range FCD). In addition to DMN-related regions, aberrant functional connectivity was also found to be present in the basal ganglia, especially in the caudate and amygdala. The FCD values of the basal ganglia (involving the caudate and amygdala) were closely related to scores from the Mini-Mental State Examination (MMSE) and the Functional Activities Questionnaire (FAQ); meanwhile, the graph theory indices (involving global efficiency and degree) of the basal ganglia (involving the caudate, amygdala, and putamen) were also found to be closely correlated with MMSE scores. In ROC analyses of both FCD and graph theory, the amygdala was of the utmost importance in the early-stage detection of MCI; additionally, the caudate nucleus was found to be crucial in the progression of cognitive decline and AD diagnosis. (4) Conclusions: It was systematically confirmed that there is a phenomenon of change in the functional connections in the basal ganglia during cognitive decline. The findings of this study could improve our understanding of AD and MCI pathology in the basal ganglia and make it possible to propose new targets for AD treatment in further studies.