In strip rolling, hydraulic automatic gauge control (HAGC) system is the key element to guarantee the precision of strip gauge. The stability of the kernel pressure closed loop (PCL) in the HAGC system plays an essential role in guaranteeing the rolling process with high performance. Nevertheless, there is some difficulty in exploring the instability mechanism of the HAGC system due to the fact that the PCL is a representative nonlinear closed-loop control system. In this work, for each component of the HAGC system, the mathematical model was established. And on the basis of the linking relation of various elements, we derived the incremental transfer model of the PCL system. Furthermore, in accordance with the deduced information transfer relation, the transfer block diagram of disturbing variable of the PCL system was obtained. Moreover, for the purpose of deriving the instability condition of the PCL system, the Popov frequency criterion was employed. The instability conditions of the HAGC system were obtained under PCL control. Furthermore, the derived instability conditions of the HAGC system were experimentally verified under various working conditions. The research results provide a fundamental foundation for studying the instability mechanism of the HAGC system.