Entrained air voids can improve the freeze-thaw durability of concrete, and also affect its mechanical and transport properties. Therefore, it is important to measure the air void structure and understand its influence on concrete performance for quality control. This paper aims to measure air void structure evolution at both early-age and hardened stages with the ultrasonic technique, and evaluates its influence on concrete properties. Three samples with different air entrainment agent content were specially prepared. The air void structure was determined with optimized inverse analysis by achieving the minimum error between experimental and theoretical attenuation. The early-age sample measurement showed that the air void content with the whole size range slightly decreases with curing time. The air void size distribution of hardened samples (at Day 28) was compared with American Society for Testing and Materials (ASTM) C457 test results. The air void size distribution with different amount of air entrainment agent was also favorably compared. In addition, the transport property, compressive strength, and dynamic modulus of concrete samples were also evaluated. The concrete transport decreased with the curing age, which is in accordance with the air void shrinkage. The correlation between the early-age strength development and hardened dynamic modulus with the ultrasonic parameters was also evaluated. The existence of clustered air voids in the Interfacial Transition Zone (ITZ) area was found to cause severe compressive strength loss. The results indicated that this developed ultrasonic technique has potential in air void size distribution measurement, and demonstrated the influence of air void structure evolution on concrete properties during both early-age and hardened stages.