Improving green total factor productivity (GTFP) is the inherent requirement for practicing the philosophy of green development and achieving regional high-quality development. Based on panel data for 68 prefectural-level-and-above cities in the Yellow River Basin of China from 2006 to 2019, we measured their GTFPs and degrees of productive-services agglomeration using the non-radial directional distance function and industrial agglomeration index formulas, respectively. Furthermore, we empirically investigated the interactive relationship between agglomeration of productive services, industrial-structure upgrading, and GTFP using the dual fixed-effects model, the mediating-effect model, and the moderating-effect model. The findings were as follows. (1) Both specialized and diversified agglomeration of productive services significantly improved the GTFPs of cities in the Yellow River Basin, and the promoting effect of specialized agglomeration was stronger than that of diversified agglomeration. (2) The diversified agglomeration of productive services (hereinafter referred to as diversified agglomeration) made a significant contribution to GTFP in all sample cities of the Yellow River Basin, while the specialized agglomeration of productive services (hereinafter referred to as specialized agglomeration) only significantly improved GTFP in the upstream cities and had no significant effect on the midstream and downstream cities. (3) When examined according to city size, specialized agglomeration was found to have a positive impact on the GTFPs of small and medium-sized cities in the Yellow River Basin but a non-significant negative impact on large cities, while the effect of diversified agglomeration on GTFP was found not to be significant. (4) Industrial-structure upgrading played partially mediating and negative moderating roles in the process of specialized agglomeration affecting the GTFPs of cities in the Yellow River Basin, but it did not become a mediating channel and moderating factor that influenced diversified agglomeration in relation to GTFP.