Excitonic states govern the optical spectra of low-dimensional semiconductor nanomaterials and their dynamics are key for a wide range of applications, such as in solar energy harvesting and lighting. Semiconducting single-walled carbon nanotubes emerged as particularly rich model systems for one-dimensional nanomaterials and as such have been investigated intensively in the past. The exciton decay dynamics in nanotubes has been studied mainly by transient absorption and time-resolved photoluminescence spectroscopy. Since different transitions are monitored with these two techniques, developing a comprehensive model to reconcile different data sets, however, turned out to be a challenge and remarkably, a uniform description seems to remain elusive. In this work, we investigate the exciton decay dynamics in single carbon nanotubes using transient interferometric scattering and time-resolved photoluminescence microscopy with few-exciton detection sensitivity and formulate a unified microscopic model by combining unimolecular exciton decay and ultrafast exciton-exciton annihilation on a time-scale down to 200 fs.