Rock-filled concrete (RFC) is an advanced construction material that integrates high-performance self-compacting concrete (HSCC) with large rocks exceeding 300 mm, providing advantages such as reduced hydration heat and increased construction processes. The elastic modulus of RFC is a critical parameter that directly influences its structural performance, making it vital for modern construction applications that require strength and stiffness. However, there is a scientific gap in understanding the effects of rock size, shape, arrangement, and volumetric ratio on this parameter. This study investigates these factors using mesoscale finite element models (FEMs) with spherical and polyhedral rocks. The results reveal that polyhedral rocks increase the elastic modulus compared to spherical rocks, enhancing RFC’s load-bearing capacity. Additionally, a 5% increase in the elastic modulus was observed when the rockfill ratio was increased from 50% to 60%, demonstrating a direct correlation between rock volume and mechanical performance. Furthermore, the elastic modulus rises significantly in the early stages of placement, followed by a gradual increase over time. Optimal rock sizes and a balanced mix of rock shapes allow for improved concrete flow and mechanical properties, making RFC a highly efficient material for construction. These findings offer valuable insights for designers and engineers looking to optimize RFC for structural applications.