Oxygen‐based anionic redox reactions have recently emerged as a lever to increase the capacity of Mn‐rich layered oxide cathodes in addition to the charge compensation based on cationic redox reactions for sodium‐ion batteries. Unfortunately, the irreversibility of anionic redox often aggravates irreversible structure change and poor cycling performance. Here, a stable anionic redox is achieved through substituting Na ions by Mg ions in P2‐type Na0.83Li0.25Mn0.75O2. Density functional theory (DFT) calculations reveal that Mg substitution effectively decreases the oxygen chemical potential, causing an improvement in lattice oxygen stability. Moreover, at a highly desodiated state, Mg ions that remain in the lattice and interact with O 2p orbitals can decrease the undercoordinated oxygen and the nonbonded, electron‐deficient O 2p states, facilitating the reversibility of oxygen redox. When cycled in the voltage range of 2.6–4.5 V where only anionic redox occurs for charge compensation, Na0.773Mg0.03Li0.25Mn0.75O2 presents a much better reversibility, giving a 4 times better cycle stability than that of Na0.83Li0.25Mn0.75O2. Experimentally, Na0.773Mg0.03Li0.25Mn0.75O2 exhibits a ≈1.1% volume expansion during sodium insertion/extraction, suggestive of a “zero‐strain” cathode. Overall, the work opens a new avenue for enhancing anionic reversibility of oxygen‐related Mn‐rich cathodes.