Fiber-reinforced concrete (FRC) has a great advantage in earthquake-resistant structures, as compared with regular concrete. However, there are many difficulties in the construction and maintenance of concrete structures due to the high density and easy corrosion of the steel fiber in commonly used steel FRC. With the development of polymer material science, polyvinyl alcohol (PVA) fiber has been rapidly promoted for use in FRC because of its low density, high strength, and large elongation at break value. Dynamic uniaxial compression and splitting tensile experiments of FRC with PVA fiber were carried out with two matrix strengths (i.e., C30 and C40), which were blended with PVA fibers with a length of 12 mm in different volume contents (0, 0.2, 0.4, and 0.6%), at the age of 28 days, under different strain rates (i.e., 10−5, 10−4, 10−3, and 10−2 s−1). The results show that PVA has an obvious enhancing and toughening effect on concrete, which can improve its brittle properties and residual strength. With increasing strain rate, the compressive strength, split tensile strength, and elastic modulus increase to a certain extent, while the toughness index and the peak strain decrease to a certain degree. The post-peak deformation characteristic changes from a brittle failure of sudden caving to a ductile failure with dense cracking. The effect of PVA is different when enhancing the concrete with two different matrix strengths. The lower the matrix strength, the more obvious the enhancement effect of the fiber, showing characteristics of a higher compressive strength and low split tensile strength in FRC with low strength and a smoother post-peak stress–strain curve.