We present a novel symmetry of the colored HOMFLY polynomial. It relates pairs of polynomials colored by different representations at specific values of N and generalizes the previously known "tug-the-hook" symmetry of the colored Alexander polynomial [1]. As we show, the symmetry has a superalgebra origin, which we discuss qualitatively. Our main focus are the constraints that such a property imposes on the general group-theoretical structure, namely the sl(N ) weight system, arising in the perturbative expansion of the invariant. Finally, we demonstrate its tight relation to the eigenvalue conjecture.