Three-dimensional (3D) path planning is a crucial technology for ensuring the efficient and safe flight of UAVs in complex environments. Traditional path planning algorithms often find it challenging to navigate complex obstacle environments, making it challenging to quickly identify the optimal path. To address these challenges, this paper introduces a Nutcracker Optimizer integrated with Hyperbolic Sine–Cosine (ISCHNOA). First, the exploitation process of the sinh cosh optimizer is incorporated into the foraging strategy to enhance the efficiency of nutcracker in locating high-quality food sources within the search area. Secondly, a nonlinear function is designed to improve the algorithm’s convergence speed. Finally, a sinh cosh optimizer that incorporates historical positions and dynamic factors is introduced to enhance the influence of the optimal position on the search process, thereby improving the accuracy of the nutcracker in retrieving stored food. In this paper, the performance of the ISCHNOA algorithm is tested using 14 classical benchmark test functions as well as the CEC2014 and CEC2020 suites and applied to UAV path planning models. The experimental results demonstrate that the ISCHNOA algorithm outperforms the other algorithms across the three test suites, with the total cost of the planned UAV paths being lower.