SARS-CoV-2 variants acquire mutations in the spike protein that promote immune evasion1 and affect other properties that contribute to viral fitness, such as ACE2 receptor binding and cell entry2,3. Knowledge of how mutations affect these spike phenotypes can provide insight into the current and potential future evolution of the virus. Here we use pseudovirus deep mutational scanning4 to measure how more than 9,000 mutations across the full XBB.1.5 and BA.2 spikes affect ACE2 binding, cell entry or escape from human sera. We find that mutations outside the receptor-binding domain (RBD) have meaningfully affected ACE2 binding during SARS-CoV-2 evolution. We also measure how mutations to the XBB.1.5 spike affect neutralization by serum from individuals who recently had SARS-CoV-2 infections. The strongest serum escape mutations are in the RBD at sites 357, 420, 440, 456 and 473; however, the antigenic effects of these mutations vary across individuals. We also identify strong escape mutations outside the RBD; however, many of them decrease ACE2 binding, suggesting they act by modulating RBD conformation. Notably, the growth rates of human SARS-CoV-2 clades can be explained in substantial part by the measured effects of mutations on spike phenotypes, suggesting our data could enable better prediction of viral evolution.