Automatic generation of feasible trajectory is one of the key technologies for autonomous flying of unmanned aerial vehicles (UAVs). The existing path planning methods, such as swarm intelligence algorithm and graph-based algorithm, cannot incorporate the flying time and UAV dynamic model into evolution. To overcome such disadvantages, a hierarchical trajectory optimization scheme consisted by improved particle swarm optimization (PSO) and Gauss pseudo-spectral method (GPM) is investigated in this paper. Firstly, considering that traditional GPM is sensitive to initial values, we design an improved PSO for path planning in the first layer. By introducing adaptive parameter adjustment strategy and position mutation updating strategy, the rapidity and optimality of the improved PSO is enhanced. Then in the second layer, a fitted curve based on the path waypoints generated by improved PSO is constructed and served as the initial values for GPM. Comparing with random initial values, the designed curve can significant improve GPM efficiency. A multi-segment strategy is also put forward to further improve the efficiency. Finally, with the consideration of dynamic model and state constraints, the time minimum trajectory planning for quadrotor UAVs is solved. Plenty of simulations are carried out and the results illustrate that the proposed scheme guarantees much better efficiency.