The alkali–silica reaction can shorten concrete life due to expansive pressure build-up caused by reaction by-products, resulting in cracking. Understanding the role of the aggregate, as the main reactive component, is essential for understanding the underlying mechanisms of the alkali–silica reaction and thereby reducing, or even preventing, any potential damage. The present study aims to investigate the role of petrographic studies along with accelerated tests in predicting and determining the potential reactivity of aggregates, including granite, rhyodacite, limestone, and dolomite, with different geological characteristics in concrete. This study was performed under accelerated conditions in accordance with the ASTM C1260 and ASTM C1293 test methods. The extent of the alkali–silica reaction was assessed using a range of microanalysis techniques including optical microscopy, scanning electron microscopy, energy-dispersive X-ray analysis, and X-ray powder diffraction. The results showed that a calcium-rich aggregate with only a small quantity of siliceous component but with a higher porosity and water adsorption rate can lead to degradation due to the alkali–silica reaction, while dolomite aggregate, which is commonly considered a reactive aggregate, showed no considerable expansion during the conducted tests. The results also showed that rhyodacite samples, due to their glassy texture, the existence of strained quartz and quartz with undulatory extinction, as well as the presence of weathering minerals, have a higher alkali-reactivity potential than granite samples.