The utilization of locally available concrete waste for producing recycled concrete aggregates is recognized as one of the most sustainable ways of satisfying the growing demand for concrete production. However, the quality of concrete waste depends on its origin and it may significantly differ from one concrete structure to another. Knowing the chemical composition of the parent concrete is crucial for determining or verifying the origin of the raw materials. For this reason, pre-demolition concrete waste streams need to be characterized and classified. Therefore, a new non-destructive method for determining the cement and aggregate type in hardened concrete using handheld X-ray fluorescence (hXRF) analyser is presented in this paper. The method was tested on different raw powders and on concretes containing different types of cements including CEM I 42.5 N (Portland cement), CEM II/B-V 42.5 N (Portland-fly ash cement), CEM III/B 42.5 N (GGBFS cement). Combined desktop XRF and Energy-dispersive X-ray Spectroscopy (EDS) measurements were used for the purpose of validation. The results revealed that the curing of concrete affects the results: a dried concrete surface condition was optimal for measurements since it limits the impact of the concrete surface moisture and efflorescence on characteristic element oxides, such as CaO. The effective measurement duration was 30 s. A CEM III/B 42.5 N (GGBFS)-based concrete surface was distinguished from other concretes using Al2O3, MgO and Fe2O3 as characteristic oxides. The inner layers of concrete were rich in SiO2, the oxide characteristic for the aggregate composition tested in this study. This shows that hXRF is suitable for use in concrete, provided that the concrete surface is dried and the characteristic elements are defined to ensure a distinction between different cement and aggregate types. Direct adoption of such characterization, however, requires field testing across a wide range of concrete compositions and in situ conditions.