Star-galaxy separation is a crucial step in creating target catalogues for extragalactic spectroscopic surveys. A classifier biased towards inclusivity risks including high numbers of stars, wasting fibre hours, while a more conservative classifier might overlook galaxies, compromising completeness and hence survey objectives. To avoid bias introduced by a training set in supervised methods, we employ an unsupervised machine learning approach. Using photometry from the Wide Area VISTA Extragalactic Survey (WAVES)-Wide catalogue comprising 9-band u - Ks data, we create a feature space with colours, fluxes, and apparent size information extracted by ProFound. We apply the non-linear dimensionality reduction method UMAP (Uniform Manifold Approximation and Projection) combined with the classifier hdbscan to classify stars and galaxies. Our method is verified against a baseline colour and morphological method using a truth catalogue from Gaia, SDSS, GAMA, and DESI. We correctly identify 99.75% of galaxies within the AB magnitude limit of Z = 21.2, with an F1 score of 0.9971 ± 0.0018 across the entire ground truth sample, compared to 0.9879 ± 0.0088 from the baseline method. Our method’s higher purity (0.9967 ± 0.0021) compared to the baseline (0.9795 ± 0.0172) increases efficiency, identifying 11% fewer galaxy or ambiguous sources, saving approximately 70,000 fibre hours on the 4MOST instrument. We achieve reliable classification statistics for challenging sources including quasars, compact galaxies, and low surface brightness galaxies, retrieving 92.7%, 84.6%, and 99.5% of them respectively. Angular clustering analysis validates our classifications, showing consistency with expected galaxy clustering, regardless of the baseline classification.
