In order to study the compressive deformation and energy evolution characteristics of concrete under dynamic loading, impact compression tests with impact velocities of 5, 6, and 7 m/s were carried out on concrete samples with aggregate volume ratios of 0, 32%, 37%, and 42%, respectively, using a split Hopkinson pressure bar test apparatus. The broken concrete pieces after destruction were collected and arranged. The fractal characteristics of fragmentation distribution of concrete specimens with different aggregate rates under impact were discussed, and the roughness of the fragment surface was characterized by the fractal dimension of the broken fragment and the crack surface energy was calculated. In addition, the analytical equation of the fractal dimension of the broken fragment and the crack surface energy was established. The relationship between the specimen energy absorption and the crack surface energy was compared and analyzed. The results show that the concrete specimens are mainly tensile split failure modes under different impact speeds. The fractal dimension, absorption energy, and crack surface energy all increase with the increase in impact speed and decrease with the increase in the aggregate rate. When the aggregate rate is different, the effective utilization rate of the absorbed energy is the largest when the aggregate content is 37%. The surface energy of the crack can be used to estimate the concrete dynamic intensity.