Aiming at the defects such as delamination and tearing that are prone to occur in the drilling process of Carbon Fiber Reinforced Plastic (CFRP), the finite element simulation and experimentation researches on drilling were carried out by using the Ultrasonic Vibration-Assisted Drilling (UVAD). In this paper, the kinematics characteristics of drill cutting edge during UVAD were analyzed to reveal the basic principle of UVAD. Then the most representative 0° and 90° angle lay-up CFRP of UVAD finite element simulation drilling model was established based on Hashin failure criterion and kinematics characteristics of UVAD. Finally, combined with drilling experiments, the accuracy of the finite element model was checked and the influence of process parameters on the cutting force was explored, and the removal mechanism of UVAD was revealed. The simulation results show that, compared with the Conventional Drilling (CD), the cutting force was reduced and removal modes was changed with the high-frequency impact cutting capability by UVAD. The experimental results also further demonstrate that UVAD effectively reduced the axial force and the delamination ratio. Specifically, the axial force during drilling was reduced by 5.1%~18.2%, the delamination factor at the hole outlet was decreased by 3.1%~8.2%, the drilling defects at the outlet were effectively suppressed, and the quality of the holes was significantly improved.