In reactive sputtering, the introduction of a reactive gas can lead to a hysteresis transition from metal to compounds in both the target and substrate. The hysteresis transition is characterized by a sudden change in partial pressure, sputtering rate, fraction of compound formation, etc. Therefore, stability is an important issue in the process control. In this paper, a mathematical model with variable sticking coefficients based on surface kinetics is used to study the stability of the process. The variable sticking coefficient represents different mechanisms for surface reactions from the Langmuir to precursor type. In order to facilitate the analysis, several nondimensional parameters are identified and used for formulation. Results show that an unsteady system converges to a steady state relatively fast at low inflow rates. With an eigenvalue analysis, the range of positive eigenvalues is consistent with the presence of a hysteresis loop. It is also found that when the chemical reaction on the substrate is moderate, a higher sputter yield of the compound leads to a more stable steady state at lower inflow rates. Regarding the sticking mechanism, for the type of precursors with the parameter k < 1, the compound is easier to form and saturate on the surface due to the higher default sticking coefficient and the lower operating conditions for the hysteresis transition.