One of the foremost causes of wellbore instability during drilling operations is shale swelling and hydration induced by the interaction of clay with water-based mud (WBM). Recently, the use of surfactants has received great interest for preventing shale swelling, bit-balling problems, and providing lubricity. Herein, a novel synthesized magnetic surfactant was investigated for its performance as a shale swelling inhibitor in drilling mud. The conventional WBM and magnetic surfactant mixed WBM (MS–WBM) were formulated and characterized using Fourier Transform Infrared (FTIR) and Thermogravimetric analyzer (TGA). Subsequently, the performance of 0.4 wt.% magnetic surfactant as shale swelling and clay hydration inhibitor in drilling mud was investigated by conducting linear swelling and capillary suction timer (CST) tests. Afterward, the rheological and filtration properties of the MS–WBM were measured and compared to conventional WBM. Lastly, the swelling mechanism was investigated by conducting a scanning electron microscope (SEM), zeta potential measurement, and particle size distribution analysis of bentonite-based drilling mud. Experimental results revealed that the addition of 0.4 wt.% magnetic surfactant to WBM caused a significant reduction (30%) in linear swelling. SEM analysis, contact angle measurements, and XRD analysis confirmed that the presence of magnetic surfactant provides long-term swelling inhibition via hydrophobic interaction with the bentonite particles and intercalation into bentonite clay layers. Furthermore, the inhibition effect showed an increase in fluid loss and a decrease in rheological parameters of bentonite mixed mud. Overall, the use of magnetic surfactant exhibits sterling clay swelling inhibition potential and is hereby proffered for use as a drilling fluid additive.