We present analysis of multiple Chandra and XMM-Newton spectra, separated by 9–19 years, of four of the youngest central compact objects (CCOs) with ages <2500yr: CXOU J232327.9+584842 (Cassiopeia A), CXOU J160103.1−513353 (G330.2+1.0), 1WGA J1713.4−3949 (G347.3−0.5), and XMMU J172054.5−372652 (G350.1−0.3). By fitting these spectra with thermal models, we attempt to constrain each CCO’s long-term cooling rate, composition, and magnetic field. For the CCO in Cassiopeia A, 14 measurements over 19 years indicate a decreasing temperature at a ten-year rate of 2.2 ± 0.2 or 2.8 ± 0.3 percent (1σ error) for a constant or changing X-ray absorption, respectively. We obtain cooling rate upper limits of 17 percent for CXOU J160103.1−513353 and 6 percent for XMMU J172054.5−372652. For the oldest CCO, 1WGA J1713.4−3949, its temperature seems to have increased by 4 ± 2 percent over a ten year period. Assuming each CCO’s preferred distance and an emission area that is a large fraction of the total stellar surface, a non-magnetic carbon atmosphere spectrum is a good fit to spectra of all four CCOs. If distances are larger and emission areas are somewhat smaller, then equally good spectral fits are obtained using a hydrogen atmosphere with B ≤ 7 × 1010G or B ≥ 1012G for CXOU J160103.1−513353 and B ≤ 1010G or B ≥ 1012G for XMMU J172054.5−372652 and non-magnetic hydrogen atmosphere for 1WGA J1713.4−3949. In a unified picture of CCO evolution, our results suggest most CCOs, and hence a sizable fraction of young neutron stars, have a surface magnetic field that is low early in their life but builds up over several thousand years.