Existing concrete random aggregate modeling methods (CRAMMs) have deficiencies in in the parameterization of the mesoscale pore structure. A novel CRAMM is proposed, whose pore structure is determined by the pore gradation, total porosity, sub-porosity, and pore size of each pore gradation segment. To study the influence of pore structure on the mechanical properties of concrete, 25 mesoscopic concrete specimens with the same aggregate structure but different meso-scale pore structures are constructed and subjected to uniaxial compression tests. For the first time, the influence of sub-porosity of each pore gradation segment, average pore radius (APR), pore specific surface area (PSSA), and total porosity on concrete failure process, compressive strength, peak strain, and elastic modulus were quantitatively and qualitatively analyzed. Results indicate that the pore structure makes the germination and propagation of the damage in cement mortar show obvious locality and affects the formation and expansion of macroscopic cracks. However, it does not accelerate the propagation of the damage in cement mortar from the periphery to the center of the specimen, nor does it change the phenomenon that the ITZ is more damaged than other meso-components of concrete before peak stress. Macroscopic cracks occur in the descending section of the stress–strain curve, and the sudden drops in the descending section of the stress–strain curve are often accompanied by the generation and expansion of macroscopic cracks. The quadratic polynomial, exponential, and power functions can well fit the relationship between total porosity and compressive strength and the relationship between PSSA and compressive strength. The linear, exponential, and power functions can well reflect the relationship between total porosity and compressive modulus and the relationship between compressive modulus and PSSA. For concrete specimens with the same total porosity, the elastic modulus and strength show randomness with the increase in the sub-porosity of macropores and are basically not affected by the APR. Based on the grey relational analysis, the effects of pore structure parameters on the elastic modulus and compressive strength are in the same order: total porosity > T [k1,k2] > T [k2,k3] > T [k3,k4] > T [k4,k5] > AVR > PSSA. The order of influence of the pore structure parameters on the peak strain is: T [k2,k3] > T [k1,k2] > T [k3,k4] > T [k4,k5] > APR > PSSA > total porosity.