We propose a theory of three-dimensional (anti) de Sitter gravity carrying Chan-Paton color charges. We define the theory by Chern-Simons formulation with the gauge algebra (gl 2 ⊕ gl 2 ) ⊗ u(N ), obtaining a color-decorated version of interacting spinone and spin-two fields. We also describe the theory in metric formulation and show that, among N 2 massless spin-two fields, only the singlet one plays the role of metric graviton whereas the rest behave as colored spinning matter that strongly interacts at large N . Remarkably, these colored spinning matter acts as Higgs field and generates a non-trivial potential of staircase shape. At each extremum labelled by k = 0, . . . , [ N −1 2 ], the u(N ) color gauge symmetry is spontaneously broken down to u(N − k) ⊕ u(k) and provides different (A)dS backgrounds with the cosmological constants N N −2k 2 Λ. When this symmetry breaking takes place, the spin-two Goldstone modes combine with (or are eaten by) the spin-one gauge fields to become partially-massless spin-two fields. We discuss various aspects of this theory and highlight physical implications.