Context. The cosmic far-infrared background (CIB) at wavelengths around 160 μm corresponds to the peak intensity of the whole extragalactic background light, which is being measured with increasing accuracy. However, the build up of the CIB emission as a function of redshift is still not well known. Aims. Our goal is to measure the CIB history at 70 μm and 160 μm at different redshifts, and provide constraints for infrared galaxy evolution models. Methods. We used deep Spitzer 24 μm catalogs complete to about 80 μJy with spectroscopic and photometric redshift identifications, derived using the GOODS and COSMOS deep infrared surveys covering 2 square degrees total. After cleaning the Spitzer/MIPS 70 μm and 160 μm maps of detected sources, we stacked the far-IR images at the positions of the 24 μm sources in different redshift bins. We measured the contribution of each stacked source to the total 70 and 160 μm light, and compared with model predictions and far-IR measurements obtained for Herschel/PACS data of smaller fields. Results. We detect components of the 70 and 160 μm backgrounds in different redshift bins up to z ∼ 2. The contribution to the CIB reaches a maximum at 0.3 ≤ z ≤ 0.9 at 160 μm (and z ≤ 0.5 at 70 μm). A total of 81% (74%) of the 70 (160) μm background was emitted at z < 1. We estimate that the AGN contribution to the far-IR CIB is less than about 10% at z < 1.5. We provide a comprehensive view of the CIB buildup at 24, 70, 100 and 160 μm. Conclusions. We find that IR galaxy models predicting a major contribution to the CIB from sources at z < 1 agree with our measurements, while our results exclude other models that predict a peak of the background at higher redshifts. The consistency of our results with those obtained by the direct study of Herschel far-IR data at 160 μm confirms that the stacking analysis method is a valid approach to estimate the components of the far-IR background using prior information about resolved mid-IR sources.