Aging and mild cognitive impairment (MCI) are accompanied by decline of cognitive functions. Meanwhile, the most common form of dementia is Alzheimer’s disease (AD), which is characterized by loss of memory and other intellectual abilities serious to make difficulties for patients in their daily life. MCI is a transition period between normal aging and dementia, which has been used for early detection of emerging dementia. It converts to dementia with an annual rate of 5–15% as compared to normal aging with 1% rate. Small decreases in the conversion rate of MCI to AD might significantly reduce the prevalence of dementia. Thus, it is important to intervene at the preclinical stage. Since there are still no effective drugs to treat AD, non-drug intervention is crucial for the prevention and treatment of cognitive decline in aging and MCI populations. Previous studies have found some cognitive brain networks disrupted in aging and MCI population, and physical exercise (PE) could effectively remediate the function of these brain networks. Understanding the exercise-related mechanisms is crucial to design efficient and effective PE programs for treatment/intervention of cognitive decline. In this review, we provide an overview of the neuroimaging studies on physical training in normal aging and MCI to identify the potential mechanisms underlying current physical training procedures. Studies of functional magnetic resonance imaging, electroencephalography, magnetoencephalography and positron emission tomography on brain networks were all included. Based on our review, the default mode network, fronto-parietal network and fronto-executive network are probably the three most valuable targets for efficiency evaluation of interventions.