The primate lentiviral accessory protein Nef downregulates CD4 and major histocompatibility complex class I (MHC-I) from the cell surface via independent endosomal trafficking pathways to promote viral pathogenesis. In addition, Nef antagonizes a novel restriction factor, SERINC5 (Ser5), to increase viral infectivity. To explore the molecular mechanism of Ser5 antagonism by Nef, we determined how Nef affects Ser5 expression and intracellular trafficking in comparison to CD4 and MHC-I. We confirm that Nef excludes Ser5 from human immunodeficiency virus type 1 (HIV-1) virions by downregulating its cell surface expression via similar functional motifs required for CD4 downregulation. We find that Nef decreases both Ser5 and CD4 expression at steady-state levels, which are rescued by NHCl or bafilomycin A1 treatment. Nef binding to Ser5 was detected in living cells using a bimolecular fluorescence complementation assay, where Nef membrane association is required for interaction. In addition, Nef triggers rapid Ser5 internalization via receptor-mediated endocytosis and relocalizes Ser5 to Rab5 early, Rab7 late, and Rab11 recycling endosomes. Manipulation of AP-2, Rab5, Rab7, and Rab11 expression levels affects the Nef-dependent Ser5 and CD4 downregulation. Moreover, although Nef does not promote Ser5 polyubiquitination, Ser5 downregulation relies on the ubiquitination pathway, and both K48- and K63-specific ubiquitin linkages are required for the downregulation. Finally, Nef promotes Ser5 colocalization with LAMP1, which is enhanced by bafilomycin A1 treatment, suggesting that Ser5 is targeted to lysosomes for destruction. We conclude that Nef uses a similar mechanism to downregulate Ser5 and CD4, which sorts Ser5 into a point-of-no-return degradative pathway to counteract its restriction. Human immunodeficiency virus (HIV) and simian immunodeficiency virus (SIV) express an accessory protein called Nef to promote viral pathogenesis. Nef drives immune escape through downregulation of CD4 and MHC-I from the host cell surface. Recently, Nef was reported to counteract a novel host restriction factor, Ser5, to increase viral infectivity. Nef downregulates cell surface Ser5, thus preventing its incorporation into virus particles, resulting in disruption of its antiviral activity. Here, we report mechanistic studies of Nef-mediated Ser5 downregulation in comparison to CD4 and MHC-I. We demonstrate that Nef binds directly to Ser5 in living cells and that Nef-Ser5 interaction requires Nef association with the plasma membrane. Subsequently, Nef internalizes Ser5 from the plasma membrane via receptor-mediated endocytosis, and targets ubiquitinated Ser5 to endosomes and lysosomes for destruction. Collectively, these results provide new insights into our ongoing understanding of the Nef-Ser5 arms race in HIV-1 infection.