The waste crumb rubber (WCR) and silica fume (SF), as industrial waste and byproduct, are widely used as additives in road construction materials, which can not only realize waste utilization and protect the environment but also reduce the consumption of building material resources. At present, most of the research has focused on the properties of concrete modified only by WCR or SF; there are few studies concentrated on composite modified concrete by the two, especially the study on pervious concrete (PC). This article aims to study the mechanical strength, deformability, and freeze-thaw (F-T) resistance of waste crumb rubber/silica fume composite modified pervious concrete (RSFPC). The RSFPC and control specimens were manufactured in the lab. The experiments of compressive strength, flexural strength, flexural failure strain, flexural elastic modulus, and F-T cycles were conducted. In addition, the properties of RSFPC were compared with WCR and SF single modification PC. The results indicate that the composite modification of WCR and SF can give full play to the advantages of the two materials so that the strength property and elastic modulus of PC will not be decreased and the F-T resistance and deformability can be greatly improved. Compared to a single modification group by WCR or SF, RSFPC has a more balanced and comprehensive performance, which will greatly expand the application of PC.