The AP-5 adaptor protein complex is presumed to function in membrane traffic, but so far nothing is known about its pathway or its cargo. We have used CRISPR-Cas9 to knock out the AP-5 ζ subunit gene, AP5Z1, in HeLa cells, and then analysed the phenotype by subcellular fractionation profiling and quantitative mass spectrometry. The retromer complex had an altered steady-state distribution in the knockout cells, and several Golgi proteins, including GOLIM4 and GOLM1, were depleted from vesicle-enriched fractions. Immunolocalisation showed that loss of AP-5 led to impaired retrieval of the cation-independent mannose 6-phosphate receptor (CIMPR), GOLIM4, and GOLM1 from endosomes back to the Golgi region. Knocking down the retromer complex exacerbated this phenotype. Both the CIMPR and sortilin interacted with the AP-5–associated protein SPG15 in pull-down assays, and we propose that sortilin may act as a link between Golgi proteins and the AP-5/SPG11/SPG15 complex. Together, our findings suggest that AP-5 functions in a novel sorting step out of late endosomes, acting as a backup pathway for retromer. This provides a mechanistic explanation for why mutations in AP-5/SPG11/SPG15 cause cells to accumulate aberrant endolysosomes, and highlights the role of endosome/lysosome dysfunction in the pathology of hereditary spastic paraplegia and other neurodegenerative disorders.