SERINC5 is a host restriction factor that impairs infectivity of HIV-1 and other primate lentiviruses and is counteracted by the viral accessory protein Nef. However, the importance of SERINC5 antagonism for viral replication and cytopathicity remained unclear. Here, we show that the Nef protein of the highly divergent SIVcol lineage infecting mantled guerezas (Colobus guereza) is a potent antagonist of SERINC5, although it lacks the CD4, CD3 and CD28 down-modulation activities exerted by other primate lentiviral Nefs. In addition, SIVcol Nefs decrease CXCR4 cell surface expression, suppress TCR-induced actin remodeling, and counteract Colobus but not human tetherin. Unlike HIV-1 Nef proteins, SIVcol Nef induces efficient proteasomal degradation of SERINC5 and counteracts orthologs from highly divergent vertebrate species, such as Xenopus frogs and zebrafish. A single Y86F mutation disrupts SERINC5 and tetherin antagonism but not CXCR4 down-modulation by SIVcol Nef, while mutation of a C-proximal di-leucine motif has the opposite effect. Unexpectedly, the Y86F change in SIVcol Nef had little if any effect on viral replication and CD4+ T cell depletion in preactivated human CD4+ T cells and in ex vivo infected lymphoid tissue. However, SIVcol Nef increased virion infectivity up to 10-fold and moderately increased viral replication in resting peripheral blood mononuclear cells (PBMCs) that were first infected with HIV-1 and activated three or six days later. In conclusion, SIVcol Nef lacks several activities that are conserved in other primate lentiviruses and utilizes a distinct proteasome-dependent mechanism to counteract SERINC5. Our finding that evolutionarily distinct SIVcol Nefs show potent anti-SERINC5 activity supports a relevant role of SERINC5 antagonism for viral fitness in vivo. Our results further suggest this Nef function is particularly important for virion infectivity under conditions of limited CD4+ T cell activation.