We observed the frequency pushing of the cavity resonance as a result of the coupling of the cavity field with the ground state 138Ba in a high-Q cavity. A weak probe laser propagated along the axis of a Fabry-Pérot cavity while ground-state barium atoms traversed the cavity mode perpendicularly. By operating the atom-cavity composite in the vicinity of an exceptional point, we could observe a greatly enhanced frequency shift of the cavity transmission peak, which was pushed away from the atomic resonance, resulting in up to 41±7 kHz frequency shift per atom from the cold cavity resonance. We analyzed our results by using the Maxwell-Schrödinger equation and obtained good agreement with the measurements.