The effectiveness of the inverse design method has been widely proven in previous studies, but research on the effect of different vortex designs on the performance of mixed flow pumps is relatively scarce. In this paper, the performances of models I1 and I2, which were designed with free and compound vortex designs, respectively, are compared to study the effect of different vortex designs on the energy characteristics and operational stability of mixed flow pumps. The results show that the efficiency of the compound vortex design at 0.8Qdes, 1.0Qdes, and 1.2Qdes is improved by 0.54%, 0.95%, and 5.91%, respectively, compared to that of the free vortex design, and the velocity and pressure pulsations under the design conditions are also significantly reduced. The internal flow analysis shows that the increased efficiency in the compound vortex design is mainly related to the reduction in the local entropy production from the hub to the mid-span and the wall entropy production from the mid-span to the shroud within the diffuser due to the improvement in the jet-wake structure near the hub. The increased operational stability is mainly related to the suppression of low-momentum fluid aggregation and H-S secondary flow caused by the increase in axial velocity near the hub and the spanwise uniformity of the total pressure.