Increased performance with reduced overall cost, and precise design and optimization of launch systems are critical to affordability. In this respect, the use of hybrid motors has increased to ease handling based on motor selection. In the current study, the launch vehicle’s performance is enhanced by incorporating a hybrid rocket motor into the last stage and optimized using particle swarm optimization to develop a six-degrees-of-freedom tool. This modification aims to increase payload placement flexibility, facilitate handling, and reduce costs. Thanks to the interactive subsystems within this research, this innovative study more comprehensively considers the launch vehicle trajectory design problem, allowing the simultaneous consideration of the effect of launch vehicle geometry along with other parameters in the system. In this context, the proposed method is applied to the Minotaur-I launch vehicle, and contributions of the detailed design and optimization are presented. Optimization results show that the percentage differences between these models for the original vehicle were observed to be 11.55% in velocity and 8.02% in altitude. However, there were differences of 10.06% and 48.8%, 15.8% and 23.2%, and 19.5% and 78.9% in altitudes and velocities when the center of gravity and moment of inertia changes were neglected, and constant aerodynamic coefficients were assumed, respectively. In all these cases, it was observed that the flight path angle was not close to zero. Moreover, the same mission was achieved by the launch vehicle with the optimized hybrid last stage and the propulsion performance was increased by about 7.64% based on the specific impulse and total impulse-over-weight ratio.