Lasting volume reductions in subcortical and temporal-insular cortices after premature birth suggest altered ongoing activity in these areas. We hypothesized altered fluctuations in ongoing neural excitability and activity, as measured by slowly fluctuating blood oxygenation of resting-state functional MRI (rs-fMRI), in premature born adults, with altered fluctuations being linked with underlying brain volume reductions. To investigate this hypothesis, 94 very preterm/very low birth weight (VP/VLBW) and 92 full-term born young adults underwent structural and rs-fMRI data acquisition with voxel-based morphometry and amplitude of low-frequency fluctuations (ALFF) as main outcome measure. In VP/VLBW adults, ALFF was reduced in lateral temporal cortices, and this reduction was positively associated with lower birth weight. Regions of reduced ALFF overlapped with reduced brain volume. On the one hand, ALFF reduction remained after controlling for volume loss, supporting the functional nature of ALFF reductions. On the other hand, ALFF decreases were positively associated with underlying brain volume loss, indicating a relation between structural and functional changes. Furthermore, within the VP/VLBW group, reduced ALFF was associated with reduced IQ, indicating the behavioral relevance of ALFF decreases in temporal cortices. These results demonstrate long-term impact of premature birth on ongoing BOLD fluctuations in lateral temporal cortices, which are linked with brain volume reductions. Data suggest permanently reduced fluctuations in ongoing neural excitability and activity in structurally altered lateral temporal cortices after premature birth.