Many gauge theories possess a hidden duality between color and kinematics in their on-shell scattering amplitudes. An open problem is to formulate an off-shell realization of the duality, thus manifesting a kinematic algebra. We show that 3D Chern-Simons (CS) theory in Lorenz gauge obeys off-shell color-kinematics duality. This holds both for the gauge field and the BRST ghosts, and the duality is manifest in the Feynman rules. A kinematic algebra can be formulated through a second-order differential operator (Poisson bracket) acting on the off-shell fields, and it corresponds to 3D diffeomorphisms generated by functions in Lorenz gauge. We consider several admissible double-copy constructions of CS theory with Yang-Mills theory, a higher-derivative (DF ) 2 gauge theory, or CS theory itself. To obtain non-vanishing amplitudes, we deform pure CS theory by including the maximum amount of adjoint matter that respects the on-shell duality. This gives a new formulation of an N = 4 CS-matter theory, with fields of unusual statistics. We argue that the color-stripped tree amplitudes of this theory are equivalent to those of the Gaiotto-Witten N = 4 CS theory with bi-fundamental matter. We further show that the double copy of the N = 4 CS theory with itself corresponds to maximally supersymmetric N = 8 Dirac-Born-Infeld theory.