Porous concrete is an energy absorption material, which has been widely used in civil engineering, traffic engineering and disaster reduction engineering. However, the effect of pore structure on the impact behavior of the porous concrete is lacked. In this study, a series of drop-weight impact tests were carried out on three typical types of porous concrete, i.e., concrete hollow brick (CHB), autoclaved aerated concrete (AAC) and foamed concrete (FC), to investigate the effect of pore structures on their impact behavior. For comparison, static load tests were also conducted as references. According to the damage to the samples, the developments of impact force, strain, contact stress–strain relationship and absorbed energy during drop-weight during the impact test were measured and analyzed. The results show that the ratio between the peak impact stress and compressive strength of CHB was 0.44, while that of AAC and FC increased to about 0.6, indicating that the small and uniform pore structure in AAC and FC had a higher resistance against impact load than the hollow cavity of CHB. In addition, the elastic recovery strain in AAC increased by about 0.2% and its strain at peak contact stress increased by about 160% for a comparison of CHB, implying that a small open pore structure could enhance ductility. Besides, the peak contact stress of FC was close to that of AAC during impact loading, while the strain at peak contact stress of FC increased by about 36% compared with AAC, revealing that the closed-pore structure could further enhance the deformation potential. Correspondingly, the energy absorption rates of CHB, AAC and FC were 85.9 kJ/s, 54.4 kJ/s and 49.7 kJ/s, respectively, where AAC decreased by about 58% compared with CHB, and FC decreased by about 10% compared with AAC.