Nickel-based superalloys also have super high strength at high temperatures, which are typically difficult to be processed. As the material of hydraulic parts, aircraft engines, and downhole intelligent tools, it has the characteristics of large cutting deformation, severe work hardening, and low thermal conductivity in deep hole machining, which will lead to the problem of poor straightness of the deep hole. In order to solve the problem of poor straightness of the deep hole in machining nickel-based superalloys, a three-dimensional finite element model of BTA deep hole drilling based on thermal-mechanical coupling is established, and the influence of different blade angles, rotation speeds, and feed rates on drilling force is calculated. The prediction model of drilling force is obtained based on the response surface regression analysis method, and the multi-objective optimization of blade angle, rotation speed, and feed rate are carried out by using the random direction search method. The BTA deep hole drilling experiments are carried out by using the optimized tool and process parameters, and the effects of different process parameters on the straightness of the deep hole of nickel-based superalloys are analyzed. The results of the research show that: Different rotation speeds and feed rates will lead to different degrees of strain hardening and thermal softening. This will lead to a significant difference in the value of the drilling force on the tool. The straightness deviation of the deep hole in nickel-based superalloys can be controlled by optimizing the tool structure and process parameters. The research work in this paper can provide a basis for the structural design of the BTA tool and optimization of process parameters.