For the same concrete quality, different types of coarse aggregates may result in different mechanical properties. This paper presents a study on the effect of aggregate types on the mechanical properties of two concretes, namely, geopolymer concrete (GP) and traditional Portland cement (TC) concrete. The mechanical properties were investigated through several large-scale tests. Moreover, scanning electron microscopy (SEM), energy dispersive spectroscopy (EDS), and laser scanning microscope (LSM) images were obtained to study the microstructure of tested mixes. The results revealed that the aggregate type has different effects on the mechanical properties of TC and GP, as they were behaving opposite to quartz and limestone aggregates. Microstructure analysis further confirmed the growth of well-bonded regions between the paste and aggregate in the GP with limestone aggregates, and the formation of several weak interfacial zones in concrete mixtures made with quartz aggregates. It was concluded that the mechanical properties of GP are very sensitive to the stiffness of aggregate, concentrations of stress, and the physical and chemical reactions occurring in the interfacial transition zone which may lead to improved or weakened bond strength between paste and aggregates.