The system formed in ultrarelativistic heavy-ion collisions behaves as a nearly-perfect fluid. This collective behavior is probed experimentally by two-particle azimuthal correlations, which are typically averaged over the properties of one particle in each pair. In this Letter, we argue that much additional information is contained in the detailed structure of the correlation. In particular, the correlation matrix exhibits an approximate factorization in transverse momentum, which is taken as a strong evidence for the hydrodynamic picture, while deviations from the factorized form are taken as a signal of intrinsic, "nonflow" correlations. We show that hydrodynamics in fact predicts factorization breaking as a natural consequence of initial state fluctuations and averaging over events. We derive the general inequality relations that hold if flow dominates, and which are saturated if the matrix factorizes. For transverse momenta up to 5 GeV, these inequalities are satisfied in data, but not saturated. We find factorization breaking in event-by-event ideal hydrodynamic calculations that is at least as large as in data, and argue that this phenomenon opens a new window on the study of initial fluctuations.