Due to reducing the reliance of buildings on fossil fuels, Passive House (PH) is receiving more and more attention. It is important that integrated optimization of passive performance by considering energy demand, cost and thermal comfort. This paper proposed a set three-stage multi-objective optimization method that combines redundancy analysis (RDA), Gradient Boosted Decision Trees (GBDT) and Nondominated sorting genetic algorithm (NSGA-II) for PH design. The method has strong engineering applicability, by reducing the model complexity and improving efficiency. Among then, the GBDT algorithm was first applied to the passive performance optimization of buildings, which is used to build meta-models of building performance. Compared with the commonly used meta-model, the proposed models demonstrate superior robustness with the standard deviation at 0.048. The optimization results show that the energy-saving rate is about 88.2% and the improvement of thermal comfort is about 37.8% as compared to the base-case building. The economic analysis, the payback period were used to integrate initial investment and operating costs, the minimum payback period and uncomfortable level of Pareto frontier solution are 0.48 years and 13.1%, respectively. This study provides the architects rich and valuable information about the effects of the parameters on the different building performance.