Resting-state functional connectivity is used throughout neuroscience to study brain organization and to generate biomarkers of development, disease, and cognition. The processes that give rise to correlated activity are, however, poorly understood. Here, we decompose resting-state functional connectivity using a "temporal unwrapping" procedure to assess the contributions of moment-tomoment activity co-fluctuations to the overall connectivity pattern. This approach temporally resolves functional connectivity at a timescale of single frames, which enables us to make direct comparisons of co-fluctuations of network organization with fluctuations in the BOLD time series. We show that, surprisingly, only a small fraction of frames exhibiting the strongest co-fluctuation amplitude are required to explain a significant fraction of variance in the overall pattern of connection weights as well as the network's modular structure. These frames coincide with frames of high BOLD activity amplitude, corresponding to activity patterns that are remarkably consistent across individuals and identify fluctuations in default mode and control network activity as the primary driver of resting-state functional connectivity. Our approach reveals fine-scale temporal structure of resting-state functional connectivity, and discloses that frame-wise contributions vary across time. These observations illuminate the relation of brain activity to functional connectivity and open a number of new directions for future research.