Laser-induced breakdown spectroscopy (LIBS) is a technique which enables the analysis of material components with precision and spatial resolution. Furthermore, the investigation method is comparatively fast which enables illustrating the distribution of elements within the examined material. This opens new possibilities for the investigation of very heterogeneous materials, such as concrete. Concrete consists of cement, water, and aggregates. As most of the transport processes take place exclusively in the hardened cement paste, relevant limit values linked to harmful element contents are specified in relation to the cement mass. When a concrete sample from an existing structure is examined, information on the concrete composition is usually not available. Therefore, assumptions have to be made to convert the element content analyzed in the sample based on the cement content in the sample. This inevitably leads to inaccuracies. Therefore, a method for distinction between cement paste and aggregates is required. Cement and aggregate components are chemically very close to each other and therefore, complex for classification. This is why the consideration of a single distinguishing feature is not sufficient. In this paper, a machine learning method is described and has been used to automate the distinction of the cement paste and aggregates of the LIBS data to receive reliable information of this technique. The presented approach could potentially be employed for many heterogeneous materials with the same complexity to quantify the arbitrary substances.