It is unclear whether the decline in executive function (EF) and perceptual-motor function (PMF) found in older adults with mild cognitive impairment (MCI) is the result of a normal aging process or due to MCI. This study aimed to determine age-related and MCI-related cognitive impairments of the EF and PMF. The EF and PMF were investigated across four groups of 240 participants, 60 in each group, including early adult, middle adult, older adult, and older adult with probable MCI. The EF, working memory, inhibition, and cognitive flexibility were evaluated using digit span backward tasks, the Stroop color-word test, and the modified switching verbal fluency test, respectively. The PMF, visual perception, visuoconstructional reasoning, and perceptual-motor coordination were evaluated using the clock reading test, stick design test, and stick catching test, respectively. Group differences were found for all subdomains of EF and PMF (p < 0.05), except for perceptual-motor coordination, indicating that this subdomain could be maintained in older adults and was not affected by MCI. For the age difference, working memory, cognitive flexibility, visual perception, and visuoconstructional reasoning remained stable across middle adults and started to decline in older adults, while cognitive inhibition began to decrease in middle adults and it further declined in older adults. To control the confounding effect of education level, the results showed that only cognitive flexibility was further decreased in older adults with probable MCI compared to those without MCI (p < 0.05). In conclusion, cognitive inhibition decreased earlier in middle adults, whereas EF and PMF started to decline in older adults. Cognitive flexibility was the only MCI-sensitive cognitive function.