With growing interest in the spatial dimension of light, multimode fibers, which support eigenmodes with unique spatial and polarization attributes, have experienced resurgent attention. Exploiting this spatial diversity often requires robust modes during propagation, which, in realistic fibers, experience perturbations such as bends and path redirections. By isolating the effects of different perturbations an optical fiber experiences, we study the fundamental characteristics that distinguish the propagation stability of different spatial modes. Fiber perturbations can be cast in terms of the angular momentum they impart on light. Hence, the angular momentum content of eigenmodes (including their polarization states) plays a crucial role in how different modes are affected by fiber perturbations. We show that, accounting for common fiber-deployment conditions, including the more subtle effect of light’s path memory arising from geometric Pancharatnam–Berry phases, circularly polarized orbital angular momentum modes are the most stable eigenbasis for light propagation in suitably designed fibers. Aided by this stability, we show a controllable, wavelength-agnostic means of tailoring light’s phase due to its geometric phase arising from path memory effects. We expect that these findings will help inform the optimal modal basis to use in the variety of applications that envisage using higher-order modes of optical fibers.