In recent years, the deterministic design optimization method has been widely used to improve the output performance of brushless direct current (BLDC) motors. However, it does not contribute to reducing the failure rate and performance variation of products because it cannot determine the manufacturing uncertainty. In this study, we proposed reliability-based robust design optimization to improve the output torque of a BLDC motor while reducing the failure rate and performance variation. We calculated the output torque and vibration response of the BLDC motor using the electromagnetic–structural coupled analysis. We selected the tooth thickness, slot opening width, slot radius, slot depth, tooth width, magnet thickness, and magnet length as the design variables related to the shape of the stator and rotor that affect the output torque. We considered the distribution of design variables with manufacturing tolerances. We performed a reliability analysis of the BLDC motor considering the distribution of design variables with manufacturing tolerances. Using the reliability analysis results, we performed reliability-based robust design optimization (RBRDO) to maximize the output torque; consequently, the output torque increased by 8.8% compared to the initial BLDC motor, the standard deviation in output performance decreased by 46.9% with improved robustness, and the failure rate decreased by 99.2% with enhanced reliability. The proposed reliability-based robust design optimization is considered to be useful in the actual product design field because it can evaluate both the reliability and robustness of the product and improve its performance in the design stage.