As a renewable resource, rape straw has rich fibrous tissue on the surface and woody parts, which can provide high toughness and tensile strength. However, rape straw is usually treated by on-site incineration, causing severe damage to the ecological environment and resources. Rape straw fiber (RSF) is added into fly ash concrete (FAC) to reuse rape straw effectively, protect the environment, and develop green and renewable building materials. In this test, RSF is divided into 4 volume additions of 0.1%, 0.2%, 0.3%, and 0.4% and 3 length intervals of 20–30, 30–40, and 40–50 mm to study the mechanical properties and microstructure of FAC. Results showed that RSF with a volume content of 0.1%–0.2% and a length of 30–40 mm can effectively improve the mechanical properties of FAC. Still, the excessive incorporation of large RSF can reduce the compressive strength, splitting tensile strength, and ultrasonic velocity of FAC. The compressive strength, splitting tensile strength, and sound velocity of specimens with a volume content of 0.2% and size length of 30–40 mm increased by 8.73%, 11.37%, and 7.60%, respectively, compared with those of the control group. Scanning electron microscopy results showed that RSF mixed with the appropriate amount of FAC presented the absence of agglomeration, the presence of bifurcation cracking, and other phenomena, and the ability to combine closely with FAC to fill internal pores and improve the mechanical properties of the FAC effectively.