Integrated modeling and operation optimization of building energy systems is significant for improving the energy utilization efficiency and reducing carbon emission. This paper introduces the standardized thermal resistance to construct an overall heat current model of the building cooling system with coupled heat transfer, mass transfer, and energy conversion processes. Based on the heat current model, we derive the holistic thermal energy transfer and conversion constraints on the system level and reduce the intermediate parameters of the system model. Moreover, the genetic algorithm is introduced to optimize the system operation conditions under the given system structure parameters. The optimization results provide the optimal mass flow distribution of cooling water, return water, and ambient air and meanwhile show that the compressor power consumption can reach 76.5% of the total system power consumption. The change of user behavior by raising the room temperature to 4°C can reduce the total system power consumption by 20%. The results are in line with the theoretical reality and prove the feasibility and effectiveness of the method proposed in this paper, which provides a practical reference for the energy‐saving operation of the building cooling system.