A deficient interferon response to SARS-CoV-2 infection has been implicated as a determinant of severe COVID-19. To identify the molecular effectors that govern interferon control of SARS-CoV-2 infection, we conducted a large-scale gain-of-function analysis that evaluated the impact of human interferon stimulated genes (ISGs) on viral replication. A limited subset of ISGs were found to control viral infection, including endosomal factors that inhibited viral entry, nucleic acid binding proteins that suppressed viral RNA synthesis, and a highly enriched cluster of ER and Golgi-resident ISGs that inhibited viral translation and egress. These included the type II integral membrane protein BST2/tetherin, which was found to impede viral release, and is targeted for immune evasion by SARS-CoV-2 Orf7a protein. Overall, these data define the molecular basis of early innate immune control of viral infection, which will facilitate the understanding of host determinants that impact disease severity and offer potential therapeutic strategies for COVID-19.