We show that an isometric action of a compact quantum group on the underlying geodesic metric space of a compact connected Riemannian manifold (M, g) with strictly negative curvature is automatically classical, in the sense that it factors through the action of the isometry group of (M, g). This partially answers a question by D. Goswami.