Based on the microdata of Chinese listed enterprises from 2012 to 2017, we empirically examine the nonlinear relationship between institutional ownership and enterprise innovation. We find that institutional ownership has an inverted-U nonlinear relationship with enterprise patenting when enterprise characteristics and fixed effects such as time, region, industry type and ownership type of enterprises (SOE or NSOE) are controlled and the endogeneity is considered. This nonlinear relationship significantly varies among active, passive, pressure-resistant and pressure-sensitive groupings of institutional investors. Employing the mediation model proposed by Hayes and Preacher (2010) [Quantifying and testing indirect effects in simple mediation models when the constituent paths are nonlinear. Multivariate Behavioral Research, 45(4), 627–660], we further investigate the mediating effects of financing constraints and R&D manipulation. The results show that institutional ownership linearly drives enterprise innovation by alleviating financing constraints but nonlinearly affects the enterprise innovation by manipulating R&D activities.