Light–matter interaction at dielectric interfaces usually manifests as spin-dependent correction to light propagation, known as classical Imbert–Fedorov (IF) shift or photonic spin Hall effect, ruled by the general spin–orbit interaction (SOI) of light. Even though vector wave equations and strong SOI-based perturbation theory in a wave picture can offer good solutions to describe the modal dispersion in optical fibers, it is difficult for all these to provide an intuitive insight into the walking off for twisted (or vortex) light beams carrying orbital angular momentum (OAM). Here we present a new perspective to the topologically spin-dependent modal splitting for the twisted light highly confined in optical fibers based on the classical IF shift on geometric optics. We verify this topologically IF-shift-based walking off by comparing the analytical results of modal splitting degrees with the solutions of eigen equation, and associate the longitudinal projection of IF shift with an interesting resonance of fiber Bragg gratings locked by the signs of SAM or OAM. This interpretation provides an insight supplement to describe light ray propagating in optical fibers together with both longitudinal Goos–Hänchen and transverse IF shift under the total internal reflection, and may benefit the development of nanoscale fiber-based light on optically classical or quantum communication and metrology.