For modern complex system, quality has become a serious and difficult problem that restricts the generation of combat effectiveness. Withal, based on domestic and foreign research and analysis, this paper puts forward a trade‐off method for complex system reliability, maintainability, supportability, testability (RMST) design, which is suitable for product design stage. First, aiming at the uncertain problem of complex system constrained optimization, the good‐point set population initialization, Deb feasibility comparison method and ε constraint method are integrated into the flower pollination algorithm, to construct the algorithm for RMST quotas design and trade‐off. Second, taking the typical series‐parallel repairable system as an example, one general model for complex system RMST quotas design and trade‐off based on effectiveness is constructed. Finally, take an unmanned nuclear radiation detection system as a typical case, to verify its feasibility, advantages, and disadvantages.