Background: Cognitive decline is age relevant and it can start as early as middle age. The decline becomes more obvious among older adults, which is highly associated with increased risk of developing dementia (e.g., Alzheimer’s disease). White matter damage was found to be related to cognitive decline through aging. The purpose of the current study was to compare the effects of Tai Chi (TC) versus walking on the brain white matter network among Chinese elderly women. Methods: A cross-sectional study was conducted where 42 healthy elderly women were included. Tai Chi practitioners (20 females, average age: 62.9 ± 2.38 years, education level 9.05 ± 1.8 years) and the matched walking participants (22 females, average age: 63.27 ± 3.58 years, educational level: 8.86 ± 2.74 years) underwent resting-state functional magnetic resonance imaging (rsfMRI) scans. Diffusion tensor imaging (DTI) and graph theory were employed to study the data, construct the white matter matrix, and compare the brain network attributes between the two groups. Results: Results from graph-based analyses showed that the small-world attributes were higher for the TC group than for the walking group (p < 0.05, Cohen’s d = 1.534). Some effects were significant (p < 0.001) with very large effect sizes. Meanwhile, the aggregation coefficient and local efficiency attributes were also higher for the TC group than for the walking group (p > 0.05). However, no significant difference was found between the two groups in node attributes and edge analysis. Conclusion: Regular TC training is more conducive to optimize the brain functioning and networking of the elderly. The results of the current study help to identify the mechanisms underlying the cognitive protective effects of TC.