Spectral unmixing of geological mixtures, such as rocks, is a challenging inversion problem because of nonlinear interactions of light with the intimately mixed minerals at a microscopic scale. The fine-scale mixing of minerals in rocks limits the sensor’s ability to identify pure mineral endmembers and spectrally resolve these constituents within a given spatial resolution. In this study, we attempt to model the spectral unmixing of two rocks, namely, serpentinite and granite, by acquiring their hyperspectral images in a controlled environment, having uniform illumination, using a laboratory-based imaging spectroradiometer. The endmember spectra of each rock were identified by comparing a limited set of pure hyperspectral image pixels with the constituent minerals of the rocks based on their diagnostic spectral features. A series of spectral unmixing paradigms for explaining geological mixtures, including those ranging from simple physics-based light interaction models (linear, bilinear, and polynomial models) to classification-based models (support vector machines (SVMs) and half Siamese network (HSN)), were tested to estimate the fractional abundances of the endmembers at each pixel position of the image. The analysis of the results of the spectral unmixing algorithms using the ground truth abundance maps and actual mineralogical composition of the rock samples (estimated using X-ray diffraction (XRD) analysis) indicate a better performance of the pure pixel-guided HSN model in comparison to the linear, bilinear, polynomial, and SVM-based unmixing approaches. The HSN-based approach yielded reduced errors of abundance estimation, image reconstruction, and mineralogical composition for serpentinite and granite. With its ability to train using limited pure pixels, the half-Siamese network model has a scope for spectrally unmixing rock samples of varying mineralogical composition and grain sizes. Hence, HSN-based approaches effectively address the modelling of nonlinear mixing in geological mixtures.