Studies of stroke have identified local reorganization in perilesional tissue. However, because the brain is highly networked, strokes also broadly alter the brain's global network organization. Here, we assess brain network structure longitudinally in adult stroke patients using resting state fMRI. The topology and boundaries of cortical regions remain grossly unchanged across recovery. In contrast, the modularity of brain systems i.e. the degree of integration within and segregation between networks, was significantly reduced sub-acutely (n = 107), but partially recovered by 3 months (n = 85), and 1 year (n = 67). Importantly, network recovery correlated with recovery from language, spatial memory, and attention deficits, but not motor or visual deficits. Finally, in-depth single subject analyses were conducted using tools for visualization of changes in brain networks over time. This exploration indicated that changes in modularity during successful recovery reflect specific alterations in the relationships between different networks. For example, in a patient with left temporo-parietal stroke and severe aphasia, sub-acute loss of modularity reflected loss of association between frontal and temporo-parietal regions bi-hemispherically across multiple modules. These long-distance connections then returned over time, paralleling aphasia recovery. This work establishes the potential importance of normalization of large-scale modular brain systems in stroke recovery.