We study the cooperative optimization problem (COP) of multi-agent systems with globally coupled cost function and coupled constraints, and design a distributed computing framework combining potential game theory underlying geometric projection. This design framework has the advantage of being able to solve the COP with globally coupled cost function and coupled constraints in distributed way. Firstly, the studied COP with coupled constraints is converted to an unconstrained one by using barrier and penalty methods, respectively, and then the coupled cost function of the unconstrained COP with n variables to be optimized is decoupled by projecting it to n hyperplanes, and n decoupled sub-optimization problems are established. Underlying this design, we exploit an equivalently changing relationship during the optimizing process between each decoupled cost function and the original global function in a fixed communication topology, which forms a potential game, and derive that the optimal solution of the COP is equivalent with Nash equilibrium of the potential game. The obtained sub-optimization problems can be solved in distributed manners and two improved distributed gradient algorithms are proposed. Finally, the distributed design is applied to the economic dispatch problem in power system to verify the superiority of the proposed algorithms.INDEX TERMS Cooperative optimization, coupled inequality constraint, potential game, barrier method, penalty method.