We study a new class of infinite-dimensional Lie algebras W ∞ (N + , N − ) generalizing the standard W ∞ algebra, viewed as a tensor operator algebra of SU (1, 1) in a grouptheoretic framework. Here we interpret W ∞ (N + , N − ) either as an infinite continuation of the pseudo-unitary symmetry U (N + , N − ), or as a "higher-U (N + , N − )-spin extension" of the diffeomorphism algebra diff(N + , N − ) of the N = N + + N − torus U (1) N . We highlight this higher-spin structure of W ∞ (N + , N − ) by developing the representation theory of U (N + , N − ) (discrete series), calculating higher-spin representations, coherent states and deriving Kähler structures on flag manifolds. They are essential ingredients to define operator symbols and to infer a geometric pathway between these generalized W ∞ symmetries and algebras of symbols of U (N + , N − )-tensor operators. Classical limits (Poisson brackets on flag manifolds) and quantum (Moyal) deformations are also discussed. As potential applications, we comment on the formulation of diffeomorphism-invariant gauge field theories, like gauge theories of higher-extended objects, and non-linear sigma models on flag manifolds.