The influence of geopolymer binder characteristics on the performance of geopolymer concrete has been extensively investigated. Yet, the influence of aggregate properties has not been thoroughly studied, and it is usually assumed that their effect is the same as in cement concrete. This study investigates the effect of aggregate on the performance of fly-ash-based geopolymer concrete. A systematic experimental study was undertaken to investigate the effect of aggregate parameters, including volume fractions (AVFs), coarse aggregate to the total aggregate ratio (CAR), maximum coarse aggregate size (MAS), and fineness modulus of fine aggregate (FFM) on the compressive strength, slump, apparent volume of permeable pores (AVPPs), and the air content of geopolymer concrete. Response surface methodology (RSM) using the central composite design approach was utilized to design the experiments and analyze the results statistically. The analysis shows that all of the investigated aggregate parameters have significant first-order effects on the measured properties. No significant interaction between any of the investigated parameters was found. The aggregate may alter the geopolymerization processes, whereby SEM-EDS analysis revealed statistically significant variations in the elemental concentrations of the produced paste as the aggregate parameters changed. Quantitative weights were assigned to the effect of the investigated aggregate parameters on the measured properties. Multi-objective optimization was carried out to obtain the best combinations of the investigated parameters. Additionally, the developed contour graphs may provide an effective tool that can be used as a guide in establishing the first trial mixtures. A wide range of consistencies (10–210 mm slump) and compressive strengths (15–55 MPa) can be obtained by controlling the aggregate grading and proportions.