Building energy efficiency, which is critical in reducing environmental impact, has become one of the most important objectives of building designs. In order to precisely express the goals of building designs, and help decision makers estimate the ultimate performance of design schemes in advance when searching for the optimal building design, the Goal Programming Model (GPM) is introduced in this study to provide a solution for explicit design objective delivery and multi-stakeholder involved decision-making support. In this proposed method, EnergyPlusTM works as a simulation engine to search for the relationship between design parameter combinations and building energy consumption. Simultaneously, Genetic Algorithm (GA) is used to improve the efficiency of overall building energy performance optimization by processing multiple iterations. A case study with five possible design scenarios was dedicated in this study to implement the proposed optimization method, and the optimization results verified the capacity of the established GP-based optimization method to satisfy various design requirements for decision makers and/or stakeholders, especially in facing the hierarchical objectives with different priorities. In this case, the envelope-related variables, including the exterior wall and window, serve as optimization objectives. The optimization is carried out under the ideal air conditioning system, considering different energy usage patterns. Meanwhile, comparing with the vague and restricted expression of objectives in multi-objective optimization, the proposed GP-based optimization method provides explicit trade-off relationships among various objectives for designers, which improves the practical value of the optimized designs, so as to ensure the project success and facilitate the development of green buildings.