Making construction waste into raw materials for recycled concrete is beneficial for resource conservation and environmental protection. This paper investigated the effects of different recycled brick aggregate (RBA) replacement rates (30%, 50%, 70%, and 100%) and different contents of polypropylene fibers (PPFs) (0.08%, 0.10%, 0.12%, 0.16%, and 0.2%) on the mechanical properties of recycled brick concrete. Gray correlation was also used to analyze the degree of effect factors on the mechanical properties of concrete. The results showed that the mechanical properties decreased when the natural coarse aggregate (NCA) was replaced with RBA, while PPFs could better improve the mechanical properties of RBA concrete. The improvement of compressive and flexural properties was optimal when the PPF content was 0.12%; the improvement of tensile properties was optimal when the PPF content was 0.2%. In addition, PPFs significantly improved the toughness of RBA concrete. The gray correlation degrees between compressive strength (tensile strength, flexural strength) and NCA, RBA, and PPFs were 0.8964 (0.8691, 0.8935), 0.7301 (0.6530, 0.7074), and 0.5873 (0.5870, 0.5840), respectively.