Ultra-High-Performance Concrete (UHPC) and Ultra-High-Performance Fiber-Reinforced Concrete (UHPFRC) represent promising materials in the field of construction, offering exceptional strength and durability, making them ideal for the development of a wide range of infrastructure projects. One of the goals is to better understand the impact of each component of the materials on their key properties in the hardened state. This work examines the effect of the aggregate on the properties of UHPC and UHPFRC. This article provides test results for five compositions without fiber, and five compositions with 2% corrugated steel fiber. Three aggregate concentrations (0, 0.2, and 0.4 m3) and quartz sand with different maximum particle sizes (0.4 and 0.8 mm) were selected. It was found that the mechanical properties of the material, such as the steel fiber bond strength, compressive and axial tensile strength, fracture energy, and critical stress intensity factor, depend on both the concentration of the aggregate and the size of its particles. A novel mix-design parameter was proposed, which reflects the total surface area of the aggregate in the composition (Sagg,tot). The relationships between the parameter Sagg,tot and the mechanical characteristics of UHPC and UHPFRC were established. The steel fiber bond strength, axial tensile strength, and fracture energy-related parameters grew non-linearly when the parameter Sagg,tot increased. When the parameter Sagg,tot was changed from 0 to 12.38 · 103 m2, the fiber bond strength increased by 1.38 times. The axial tensile strength and total fracture energy of the UHPFRC increased by 1.48 and 1.63 times, respectively. The compressive strength changed linearly and increased by 1.12 times. The improvement in the mechanical properties of the material was associated with an increase in the friction force between the fiber and the matrix, which was confirmed by the formation of a greater number of scratches on the surface of the fiber with an increasing value of the parameter Sagg,tot. The deformation characteristics, such as modulus of elasticity, Poisson’s ratio, and drying shrinkage strain, were determined solely by the volumetric concentration of the aggregate, as in conventional concrete. An increase in the aggregate volume content from 0 to 0.4 m3 led to an increase in the modulus of elasticity of 1.41–1.44 times, and a decrease in the ultimate shrinkage strain of almost 2 times. The dependencies obtained in this work can be used to predict the properties of UHPC and UHPFRC, taking into account the type and volume concentration of the aggregate.