The low impact strength of short carbon fiber (CF) reinforced polypropylene (PP) composites (PP/CF) restricts their practical application. Many studies focus on improving interfacial adhesion strength between PP and CF, while the essential properties of PP matrix are also significant, but rarely investigated for toughening PP/CF composites. In this work, the combination of β‐modification and annealing treatment were adopted to modify the intrinsic toughness of PP matrix, thereby enhancing the impact strength of PP/CF composites, meanwhile their flexural strength and modulus were almost maintained. From the XRD, DSC, POM, DMA and SAXS results, we observed that abundant β‐crystals with small size were produced in PP matrix after β‐modification. In addition, increased overall crystallinity, perfected crystals structure and looser amorphous region were generated after annealing treatment. These microstructure changes resulting from β‐modification and annealing treatment, together triggered the plastic deformation of PP matrix, which was the main reason for toughening PP/CF composites.