Since entering the human population, SARS-CoV-2 (the causative agent of COVID-19) has spread across the world, causing >40 million infections and >1 million deaths. While large-scale sequencing efforts have identified numerous genetic mutations in SARS-CoV-2 during its circulation, it remains largely unclear whether these changes impact adaptation, replication or transmission of the virus in its new host. Here, we characterized 14 different low-passage replication-competent human SARS-CoV-2 isolates representing all the major European clades observed during the first pandemic wave in early 2020. By integrating viral sequencing data from patient material, viral stocks and passaging experiments, with kinetic virus replication data from non-human Vero-CCL81 cells and primary differentiated human bronchial epithelial cells (BEpCs), we observed several SARS-CoV-2 sequence features that associate with distinct phenotypes. Notably, naturally-occurring substitutions in Orf3a (Q57H) and nsp2 (T85I) were associated with poor replication in Vero-CCL81 cells but not in BEpCs, while SARS-CoV-2 isolates expressing the Spike D614G substitution generally exhibited enhanced replication abilities in BEpCs. Strikingly, low-passage Vero-derived stock preparation of 3 SARS-CoV-2 isolates selected for substitutions at positions 5/6 of E, and were highly attenuated in BEpCs, revealing a key cell-specific function to this region. Rare isolate-specific deletions were also observed in the Spike furin-cleavage site during Vero-CCL81 passage, but these were rapidly selected against in BEpCs, underscoring the importance of this site for SARS-CoV-2 replication in primary human respiratory cells. Overall, our study uncovers natural sequence features in the SARS-CoV-2 genome that determine efficient virus replication and tropism for the human respiratory epithelium.