Carbon fiber reinforced polymers (CFRPs) are interesting materials due to their excellent properties, such as their high strength-to-weight ratio, low thermal expansion, and high fatigue resistance. However, to meet the requirements for their assembly, the drilling processes involved should be optimized. Defects such as delamination, dimensional errors and poor internal surface finish can lead to the premature failure of parts when bolt-joined or rivet-connected. In addition, the characteristic anisotropy and heterogeneity of these materials, and the issues related to the temperature reached during drilling, make it difficult to obtain optimal cutting parameters or to achieve high material removal rates. This research focuses on the optimization of the CFRPs drilling process by means of experimental analysis—varying the feed and spindle speed—for two different types of commercial drills—a twist tool and a dagger tool. An automatic image processing methodology was developed for the evaluation of the dimensional accuracy and delamination of the holes. The optimization was carried out using a multi-objective regression technique based on the dimensional deviations, delamination and surface finish. The areas with favorable machining conditions have been delimited for both tools and the results indicate that the twist tool allows one to achieve more productive cutting conditions than the dagger tool, when the combination of low feeds and high spindle speeds are the conditions to be avoided.