China possesses abundant pumice resources and thereby makes the utilization of pumice in the preparation of pumice aggregate concrete (PAC) a significant strategy for environmental protection and resource conservation. To obtain the effect of pumice pore structure variation on the compressive strength of PAC, PACs with strength classes LC20, LC30, and LC40 were prepared. Moreover, the pore structure of PAC was characterized using nuclear magnetic resonance to investigate the effect of pore structure variation on the compressive strength of PAC. Results showed that the higher the coarse aggregate content of PAC, the higher the percentage of large capillary and non-capillary pore sizes of PAC, corresponding to higher porosity and lower compressive strength. The hydration products in PAC continuously fill in the pore structure, the proportion of large capillary pores and non-capillary pore size gradually decreases, the proportion of small capillary pores and medium capillary pore size gradually increases, the pumice concrete matrix gradually becomes dense, and the compressive strength increases. The prediction model of the pore structure and compressive strength is established based on gray theory, and the relative error between predicted and tested values is not significant, which can effectively predict its compressive strength. It provides effective guidance for the engineering practical application of PAC.