We study the effect of asymmetric fermionic dark matter (DM) on the thermal evolution of neutron stars (NSs). No interaction between DM and baryonic matter is assumed, except the gravitational one. Using the two-fluid formalism, we show that DM accumulated in the core of a star pulls inwards the outer baryonic layers of the star, increasing the baryonic density in the NS core. As a result, it significantly affects the star’s thermal evolution by triggering an early onset of the direct Urca process and modifying the photon emission from the surface caused by the decrease of the radius. Thus, due to the gravitational pull of DM, the direct Urca process becomes kinematically allowed for stars with lower masses. Based on these results, we discuss the importance of NS observations at different distances from the Galactic center. Since the DM distribution peaks towards the Galactic center, NSs in this region are expected to contain higher DM fractions that could lead to a different cooling behavior.