A trajectory optimization algorithm is developed in this article to compute a trajectory that avoids obstacles for an unmanned aerial vehicle and minimize the load factor. Quadratic and polyhedron obstacles are both considered in this article. To satisfy the safety distance requirements in engineering practice, a distance‐based obstacle avoidance constraint is formulated, which restricts the minimum distance that a vehicle can approach the obstacle. Strong duality equivalent converts the constraints into equivalent forms without introducing binary variables. Subsequently, the trajectory optimization problem is numerically solved via a hp‐pseudospectral sequential convex programming method, which iteratively solves a sequence of second‐order cone programming subproblems until convergence is attained. Furthermore, a modified trust‐region constraint and a customized nominal trajectory update rule are proposed to accelerate the convergence of the algorithm. A group of simulations are performed to demonstrate that the algorithm is computationally efficient and capable of generating a trajectory that satisfies safety distance requirements, preventing conservativeness in the trajectory, reducing the incidence of intersample constraints violation.