This study explored the effect of the long-term partial replacement of chemical fertilizer with organic fertilizer on soil organic carbon composition, enzyme activity, and crop yields in the wheat–maize rotation area of northern Anhui, China. This study also specified the proper amount of organic fertilizer replacement that should be used for chemical fertilizer. Different fertilization modes were used (no fertilization, CK; chemical fertilizer, CF; chemical fertilizer and straw returning, CF + S; chemical fertilizer, straw returning, and straw decomposition agent, CF + S + DA; 70% chemical fertilizer and 50% organic fertilizer, 70% CF + 50% OF; 70% chemical fertilizer, 50% organic fertilizer and straw returning, 70% CF + 50% OF + S; 50% chemical fertilizer and 100% organic fertilizer, 50% CF + 100% OF; and 50% chemical fertilizer, 100% organic fertilizer, and straw returning, 50% CF + 100% OF + S). Variations in the organic carbon composition, enzyme activity, soil pH, and crop yields in the wheat–maize rotation under different fertilization treatments were analyzed. The results showed that the replacement of chemical fertilizer with organic fertilizer results in improved crop yields in wheat–maize rotation. The long-term partial replacement of chemical fertilizer with organic fertilizer can increase the quality of soil humus, alleviate soil acidification, and improve soil enzyme activity. Straw returning and organic fertilizer application can considerably raise the activities of urease, acid phosphatase, and nitrate reductase in soil. The soil pH of the CF treatment was reduced compared to the CK treatment, while organic fertilizer application alleviated soil acidification when compared to CF treatment. Organic fertilization increases the total organic carbon content of the soil, which was 19.6~85.5% higher than in the CK treatment. Applying straw and organic fertilizer significantly increased the ratio of the humic/fulvic acid in the soil. The soil active carbon forms of the soil with the application of organic fertilizer and straw returning were significantly higher than those of the CK and CF treatments. This study suggests that the optimal fertilizer management option in northern Anhui’s wheat–maize rotation area is to replace 50% of the chemical fertilizer with organic fertilizer, and to fully return straw to the field. This would include 150 kg N h·m−2, 60 kg P2O5 h·m−2, 50 kg K2O h·m−2, 6000 kg organic fertilizer h·m−2, and full straw return to the field.