A molecular pump is the core component of vacuum systems in portable mass spectrometers and other analytical instruments. The forms of the existing molecular pumps mainly are the combinations of vertical bleed and compression channel, which have the shortcomings of heavy mass and large volume, which seriously restricts the application and development of portable mass spectrometers. Aiming at the problems of low strength and insufficient pumping performance under the miniaturization constraints (mass of 1.8 kg; exhaust diameter of 25 mm) of molecular pumps, a compound pump consisting of a horizontal bleed channel and multi-stage spiral compression channel is proposed. The pumping principle of the compound molecular pump is analyzed to obtain its preliminary structural size parameters. The test particle Monte Carlo method is presented for establishing an aerodynamic model for a high-speed small compound molecular pump, which can be used to investigate the pumping performance of bleed blades and compression channels in a thin air environment. On the basis of the aerodynamic model, the NNIA multi-objective optimization algorithm is presented to optimize the structural parameters of the compound molecular pump. After structural parameter optimization, the maximum flow rate and compression ratio of the compound molecular pump are increased by 13.6% and 41.6%, respectively. The experimental results of the pumping performance show that the predicted data of the aerodynamic model are in good agreement with the experimental data, with an error of 12–27%. Namely, the established aerodynamic model has high accuracy and the optimized structural parameters of the compound molecular pump can provide basic conditions for the large-scale application and promotion of portable mass spectrometers.