We discuss Transverse Momentum Dependent (TMD) gluon distributions within high energy factorization at next-to-leading order in the strong coupling within the framework of Lipatov's high energy effective action. We provide a detailed discussion of both rapidity divergences related to the TMD definition and its soft factor on the one hand, and rapidity divergences due to high energy factorization on the other hand, and discuss common features and differences between Collins-Soper (CS) and Balitsky-Fadin-Kuarev-Lipatov (BFKL) evolution. While we confirm earlier results which state that the unpolarized and linearly polarized gluon TMD agree in the BFKL limit at leading order, we find that both distributions differ, once next-to-leading order corrections are being included. Unlike previous results, our framework allows to recover the complete anomalous dimension associated with Collins-Soper-Sterman (CSS) evolution of the TMD distribution, including also single-logarithmic terms in the CSS evolution. As an additional result we provide a definition of k T -factorization, i.e. matching of off-shell coefficients to collinear factorization at next-to-leading order within high energy factorization and the effective action framework. We furthermore establish a link between the QCD operator definition of the TMD gluon distribution and a previously derived off-shell TMD gluon-to-gluon splitting function, which is within the present framework obtained as the real 1-loop correction.