Asphalt concrete is widely used in hydraulic structure facilities as an impermeable structure in alpine cold regions, and its dynamic mechanical properties are influenced by the strain rate and specimen size. However, the specimen size has an important effect on mechanical properties; few systematic studies have investigated on the size effect of hydraulic asphalt concrete (HAC) under dynamic or static loading rates. In the present study, four sizes of cylindrical roller-compacted hydraulic asphalt concrete (RCHAC) specimens with heights of 50 mm, 100 mm, 150 mm, and 200 mm were prepared and tested under different loading rates ranging from 10−5 s−1 to 10−2 s−1 to investigate the size effects of mechanical properties and failure modes at the temperature of 5 °C. The effect of strain rate on the size effects of the compressive strength and the elastic modulus of RCHAC have also been explored. These tests indicate that when the specimen size increases, the compressive strength and failure degree decrease, while the elastic modulus increases. When the height increases from 50 mm to 200 mm, the compressive strength at different strain rates decreased by more than 50%. Furthermore, the elastic modulus increased by about 211.8% from 0.51 GPa to 1.59 GPa at a strain rate of 10−5 s−1, and increased by 150% from 5.08 GPa to 12.71 GPa at a strain rate of 10−2 s−1. As the strain rate increases, the variation trends with the size of the compressive strength, elastic modulus, and failure degree are distinctly intensified. A modified dynamic size effect law, which incorporates both the specimen size and strain rate, is proposed and verified to illustrate the dynamic size effect for the RCHAC under different loading rates.