Tumor growth pattern (TGP) and perineural invasion (PNI) at the invasive margin have been recognized as indicators of tumor invasiveness and prognostic events in colorectal cancer (CRC). This study aims to develop a scoring system incorporating TGP and PNI, and further investigate its prognostic significance for CRC risk stratification. A scoring system, termed tumor-invasion score, was established by summing TGP and PNI scores. The discovery cohort (N = 444) and the validation cohort (N = 339) were used to explore the prognostic significance of the tumor-invasion score. The endpoints of the event were disease-free survival (DFS) and overall survival (OS) which were analyzed by the Cox proportional hazard model. In the discovery cohort, Cox regression analysis showed that DFS and OS were inferior for score 4 group compared with score 1 group (DFS, hazard ratio (HR) 4.44, 95% confidence interval (CI) 2.49–7.92, p < 0.001; OS, 4.41, 2.37–8.19,p < 0.001). The validation cohort showed similar results (DFS, 4.73, 2.39–9.37, p < 0.001; OS, 5.52, 2.55–12.0, p < 0.001). The model combining tumor-invasion score and clinicopathologic information showed good discrimination performance than single predictors. TGP and PNI were associated with tumor invasiveness and survival in CRC. The tumor-invasion score generated by TGP and PNI scores served as an independent prognostic parameter of DFS and OS for CRC patients.