There is an urgent need to change the economic development mode from “resources driven” to “innovation driven” with the stagnation of the economy in China. Most existing research on the effect of high-speed rail (HSR) on firm innovation has lacked theoretical support and empirical evidence of firm innovation through knowledge spillover. This study introduces HSR as a cost coefficient to the classical heterogeneous firm model to construct a theoretical framework to determine the impact of HSR on firms’ innovation output. By matching the data of listed firms with the data of prefecture-level cities, the general difference-in-differences (DID) method is used to explore the impact of HSR on firm innovation and its mechanism. The research shows that the construction of HSR has a significant effect on the number of applied patent and authorized patents of firms and that there is a marginal increasing trend relating to the density and timing of HSR. The study found that in peripheral cities, firms in industries with rapid technological advances and highly innovative behaviors benefit more from HSR. HSR is associated with knowledge spillover within and between central and peripheral cities. It also has a heterogeneous sorting effect bounded by city size that promotes highly educated talent and the innovative output of firms that becomes significant only after the population size of a city reaches a certain threshold. HSR stimulates firm innovation mainly by improving the effect of firm resource allocation, promoting the spillover effect of innovation due to the flow and aggregation of resources, and increasing the scale effect of market expansion. Therefore, when designing innovation policies, the role of improving the construction of transportation to increase the frequency of face-to-face communication should be included, thus promoting the flow of knowledge and research collaboration.