We characterize the first 40 Myr of evolution of circumstellar disks through a unified study of the infrared properties of members of young clusters and associations with ages from 2 Myr up to ∼40 Myr: NGC 1333, NGC 1960, NGC 2232, NGC 2244, NGC 2362, NGC 2547, IC 348, IC 2395, Chamaeleon I, Orion OB1a and OB1b, Taurus, the β Pictoris Moving Group, ρ Ophiuchi, and the associations of Argus, Carina, Columba, Scorpius-Centaurus, and Tucana-Horologium. Our work features: (1) a filtering technique to flag noisy backgrounds; (2) a method based on the probability distribution of deflections, P(D), to obtain statistically valid photometry for faint sources; and (3) use of the evolutionary trend of transitional disks to constrain the overall behavior of bright disks. We find that the fraction of disks three or more times brighter than the stellar photospheres at 24 μm decays relatively slowly initially and then much more rapidly by ∼10 Myr. However, there is a continuing component until ∼35 Myr, probably due primarily to massive clouds of debris generated in giant impacts during the oligarchic/chaotic growth phases of terrestrial planets. If the contribution from primordial disks is excluded, the evolution of the incidence of these oligarchic/chaotic debris disks can be described empirically by a log-normal function with the peak at 12-20 Myr, including ∼13% of the original population, and with a postpeak mean duration of 10-20 Myr.