A new self-consistent procedure for calculating the total energy with an orbital-dependent density functional approximation (DFA), the generalized optimized effective potential (GOEP), is developed in the present work. The GOEP is a nonlocal Hermitian potential that delivers the sets of occupied and virtual orbitals and minimizes the total energy. The GOEP optimization leads to the same minimum as does the orbital optimization. The GOEP method is promising as an effective optimization approach for orbital-dependent functionals, as demonstrated for the self-consistent calculations of the random phase approximation (RPA) to the correlation functionals in the particle—hole (ph) and particle—particle (pp) channels. The results show that the accuracy in describing the weakly interacting van der Waals systems is significantly improved in the self-consistent calculations. In particular, the important single excitations contribution in non-self-consistent RPA calculations can be captured self-consistently through the GOEP optimization, leading to orbital renormalization, without using the single excitations in the energy functional.