While cubic Quasi-topological gravity is unique, there is a family of quartic Quasi-topological gravities in five dimensions. These theories are defined by leading to a first order equation on spherically symmetric spacetimes, resembling the structure of the equations of Lovelock theories in higher-dimensions, and are also ghost free around AdS. Here we construct slowly rotating black holes in these theories, and show that the equations for the off-diagonal components of the metric in the cubic theory are automatically of second order, while imposing this as a restriction on the quartic theories allows to partially remove the degeneracy of these theories, leading to a threeparameter family of Lagrangians of order four in the Riemann tensor. This shows that the parallel with Lovelock theory observed on spherical symmetry, extends to the realm of slowly rotating solutions. In the quartic case, the equations for the slowly rotating black hole are obtained from a consistent, reduced action principle. These functions admit a simple integration in terms of quadratures. We also prove that beyond the slowly-rotating approximation, for generic values of the couplings, it is impossible to embed the rotating solution in a Kerr-Schild ansatz.