Human postural control system is inherently complex with nonlinear interaction among multiple subsystems. Accordingly, such postural control system has the flexibility in adaptation to complex environments. Previous studies applied complexity-based methods to analyze center of pressure (COP) to explore nonlinear dynamics of postural sway under changing environments, but direct evidence from central nervous system or muscular system is limited in the existing literature. Therefore, we assessed the fractal dimension of COP, surface electromyographic (sEMG) and electroencephalogram (EEG) signals under visual-vestibular habituation balance practice. We combined a rotating platform and a virtual reality headset to present visual-vestibular congruent or incongruent conditions. We asked participants to undergo repeated exposure to either congruent (n = 14) or incongruent condition (n = 13) five times while maintaining balance. We found repeated practice under both congruent and incongruent conditions increased the complexity of high-frequency (0.5–20 Hz) component of COP data and the complexity of sEMG data from tibialis anterior muscle. In contrast, repeated practice under conflicts decreased the complexity of low-frequency (<0.5 Hz) component of COP data and the complexity of EEG data of parietal and occipital lobes, while repeated practice under congruent environment decreased the complexity of EEG data of parietal and temporal lobes. These results suggested nonlinear dynamics of cortical activity differed after balance practice under congruent and incongruent environments. Also, we found a positive correlation (1) between the complexity of high-frequency component of COP and the complexity of sEMG signals from calf muscles, and (2) between the complexity of low-frequency component of COP and the complexity of EEG signals. These results suggested the low- or high-component of COP might be related to central or muscular adjustment of postural control, respectively.