To reduce drying shrinkage of AASC mortar (AASM), mixed aggregate mixed with river sand (RS) and silica sand in three sizes was used to investigate the effect of the physical properties of mixed aggregate on shrinkage reduction. A mixture of river sand (0.2–0.8 mm), S1 (2.5–5.0 mm), S2 (1.6–2.5 mm), and S3 (1.21–160 mm) had river sand–silica sand mean diameter ratios (dr) of 7.68 (S1/RS), 3.75 (S2/RS), and 3.02 (S3/RS). The compressive strength and drying shrinkage characteristics of mixed aggregates according to fineness modulus, surface area, bulk density, and pore space were investigated. It had the highest bulk density and lowest porosity at a substitution ratio of 50%, but the highest strength was measured at a substitution ratio of 50% or less. High mechanical properties were shown when the fineness modulus of the mixed aggregate was in the range of 2.25–3.75 and the surface area was in the range of 2.25–4.25 m2/kg. As the substitution rate of silica sand increased, drying shrinkage decreased. In particular, the drying shrinkage of RS + S1 mixed aggregate mixed with S1 silica sand, which had the largest particle size, was the smallest. When silica sand or river sand was used alone, the drying shrinkage of the sample manufactured only with S1, which has the largest particle size of silica sand, was the smallest among all mixes. Compared to RS, at a 5% activator concentration, drying shrinkage was reduced by approximately 40% for S1, 27% for S2, and 19% for S3. At a 10% concentration, S1 showed a reduction effect of 39%, S2 by 28%, and S3 by 13%. As a result of this study, it was confirmed that the drying shrinkage of AASM could be reduced simply by controlling the physical properties of the aggregate mixed with two types of aggregate. This is believed to have a synergistic effect in reducing drying shrinkage when combined with various reduction methods published in previous studies on AASM shrinkage reduction. However, additional research is needed to analyze the correlation and influencing factors between the strength, pore structure, and drying shrinkage of AASM using mixed aggregate.