Poly(ethylene terephthalate) (PET) composites containing carbon fiber (CF) or polymethyl methacrylate (PMMA)-grafted carbon fiber (PMMA-g-CF) were prepared by melt compounding. The rheology, non-isothermal crystallization behavior, and mechanical and thermal properties of pure PET, PET/CF and PET/PMMA-g-CF composites were investigated. The results show that the addition of CF or PMMA-g-CF significantly increases the storage modulus (G′), loss modulus (G″), and complex viscosity (η*) of the composites at low frequency. The Cole-Cole plots confirm that the surface modification of CF leads to a better interaction between the CF and PET, and then decreases the heterogeneity of the polymeric systems, which is confirmed by the SEM observation on the tensile fracture surface of the composites. Non-isothermal crystallization analysis shows that the CF or PMMA-g-CF could serve as nucleation agent to accelerate the crystallization rate of the composites, and the effect of PMMA-g-CF is stronger than that of CF. The result is further confirmed by the analysis of the crystallization activation energy for all composites calculated by the Flynn-Wall-Ozawa method. Moreover, the tensile and impact strength and the thermal stability of the composites are improved by CF, while the incorporation of PMMA-g-CF further enhances the tensile and impact strength and thermal stability.