A growing body of evidence clearly indicates the beneficial effects of physical activity (PA) on cognition. The importance of PA is now being reevaluated due to the increase in sedentary behavior in older adults during the COVID-19 pandemic. Although many studies in humans have revealed that PA helps to preserve brain health, the underlying mechanisms have not yet been fully elucidated. In this review, which mainly focuses on studies in humans, we comprehensively summarize the mechanisms underlying the beneficial effects of PA or exercise on brain health, particularly cognition. The most intensively studied mechanisms of the beneficial effects of PA involve an increase in brain-derived neurotrophic factor (BDNF) and preservation of brain volume, especially that of the hippocampus. Nonetheless, the mutual associations between these two factors remain unclear. For example, although BDNF presumably affects brain volume by inhibiting neuronal death and/or increasing neurogenesis, human data on this issue are scarce. It also remains to be determined whether PA modulates amyloid and tau metabolism. However, recent advances in blood-based biomarkers are expected to help elucidate the beneficial effects of PA on the brain. Clinical data suggest that PA functionally modulates cognition independently of neurodegeneration, and the mechanisms involved include modulation of functional connectivity, neuronal compensation, neuronal resource allocation, and neuronal efficiency. However, these mechanisms are as yet not fully understood. A clear understanding of the mechanisms involved could help motivate inactive persons to change their behavior. More accumulation of evidence in this field is awaited.