SARS-CoV-2 is a human pathogen that causes infection in both the upper respiratory tract (URT) and the lower respiratory tract (LRT). The viral kinetics of SARS-CoV-2 infection and how they relate to infectiousness and disease progression are not well understood. Here, we develop data-driven viral dynamic models of SARS-CoV-2 infection in both the URT and LRT. We fit the models to viral load data from patients with likely infection dates known, we estimated that infected individuals with a longer incubation period had lower rates of viral growth, took longer to reach peak viremia in the URT, and had higher chances of presymptomatic transmission. We then developed a model linking viral load to infectiousness. We found that to explain the substantial fraction of transmissions occurring presymptomatically, the infectiousness of a person should depend on a saturating function of the viral load, making the logarithm of the URT viral load a better surrogate of infectiousness than the viral load itself. Comparing the roles of target-cell limitation, the innate immune response, proliferation of target cells and spatial infection in the LRT, we found that spatial dissemination in the lungs is likely to be an important process in sustaining the prolonged high viral loads. Overall, our models provide a quantitative framework for predicting how SARS-CoV-2 within-host dynamics determine infectiousness and represent a step towards quantifying how viral load dynamics and the immune responses determine disease severity.