Basalt fiber-reinforced pea gravel concrete (BFRPGC) has remarkable potential for use as the retrofitting covers for masonry walls. However, a quantitative understanding of the mechanical properties of the BFRPGC material is still a perceived gap in the current literature. In this study, the role of basalt fibers in pea gravel concrete was evaluated by a comprehensive experimental investigation involving compressive strength tests and splitting tensile tests. Fiber length and volume fraction were selected as the key parameters. Two fiber lengths of 6 mm and 12 mm were considered, while the volume fraction corresponding to each of the fiber lengths was increased from 0.3% to 0.8%, with a step of 0.1%. The measured strengths were not simply proportional to the fiber volume fraction. The reason behind this phenomenon, i.e., the coupling effect of the bridging role of basalt fibers on concrete microcracks and the fiber agglomeration in concrete, was analyzed. The best performance of the BFRPGC material was achieved by incorporating 12-millimeter-length fibers with a volume fraction of 0.4%. Compared to that of the reference pea gravel concrete, a significant increase of up to 44.5% in compressive strength was recorded in this case. Furthermore, the failure mechanism of basalt fibers in pea gravel concrete was disclosed via the scanning electron microscope observations. In addition, the uniaxial compressive stress–strain model of the BFRPGC material was established.