The duality between color and kinematics was originally observed for purely adjoint massless gauge theories, and later found to hold even after introducing massive fermionic and scalar matter in arbitrary gauge-group representations. Such a generalization was critical for obtaining both loop amplitudes in pure Einstein gravity and realistic gravitational matter from the double copy. In this paper we elaborate on the double copy that yields amplitudes in gravitational theories coupled to flavored massive matter with spin, which is relevant to the problems of black-hole scattering and gravitational waves. Our construction benefits from making the little group explicit for the massive particles, as shown on lower-point examples. For concreteness, we focus on the double copy of QCD with massive quarks, for which we work out the gravitational Lagrangian up to quartic scalar and vector-scalar couplings. We find new gauge-invariant double-copy formulae for tree-level amplitudes with two distinct-flavor pairs of matter and any number of gravitons. These are similar to, but inherently different from, the well-known Kawai-Lewellen-Tye formulae, since the latter only hold for the double copy of purely adjoint gauge theories.