Carbon fiber reinforced polymer (CFRP) composites are widely used in various fields because of their high strength, good toughness, and low density. However, owing to their unique forming process, some complex structures such as holes and grooves cannot be formed directly. Therefore, traditional machining procedures are also required. The drilling process is one of the most common machining methods for CFRP material holes, but owing to the complex structure and difficulty in processing CFRP composite material, the quality of the drilling process is often challenging to guarantee. Moreover, the hole-forming defects also have complex forms and lack uniform evaluation indexes. This study summarizes the common orifice defects in the process of CFRP drilling, establishes a comprehensive evaluation method of orifice defects by introducing the relevant theory of the fuzzy complementary judgment matrix, and experimentally verifies the accuracy and reliability of this method. Then, based on the experimental results, a non-linear cutting parameter optimization model is established, which effectively suppresses the orifice defects to ensure the accuracy of the hole size, roundness, and hole wall roughness. The hole-forming quality is subsequently improved. The hole quality evaluation method proposed in this study reduced the dimension of the evaluation index to ensure relevance and effectiveness and improved the convenience of quality inspection and parameter optimization in actual production.