The incorporation of a recycled concrete aggregate (RCA) as a replacement of natural aggregates (NA) in road construction has been the subject of recent research. This tendency promotes sustainability, but its use depends mainly on the final product’s properties, such as chemical stability. This study evaluates the physical and chemical properties of RCAs from two different sources in comparison with the performance of NA. One RCA was obtained from the demolition of a building (recycled concrete aggregate of a building—RCAB) and another RCA from the rehabilitation of a Portland cement concrete pavement (recycled concrete aggregate from a pavement—RCAP). Characterization techniques such as X-ray fluorescence (XRF), X-ray diffraction (XRD), UV spectroscopy, and atomic absorption spectrometry were used to evaluate the RCAs’ coarse fractions for chemical potential effects on asphalt mixtures. NA was replaced with RCA at 15%, 30%, and 45% for each size of the coarse fractions (retained 19.0, 12.5, 9.5, and 4.75 sieves in mm). The mineralogical characterization results indicated the presence of quartz (SiO2) and calcite (CaCO3) as the most significant constituents of the aggregates. XFR showed that RCAs have lower levels of CaO and Al2O3 concerning NA. Potential reactions in asphalt mixtures by nitration, sulfonation, amination of organic compounds, and reactions by alkaline activation in the aggregates were discarded due to the minimum concentration of components such as NO2, (–SO3H), (–SO2Cl), and (Na) in the aggregates. Finally, this research concludes that studied RCAs might be used as replacements of coarse aggregate in asphalt mixtures since chemical properties do not affect the overall chemical stability of the asphalt mixture.