Most maintenance optimization models of gear systems have considered single failure mode. There have been very few papers dealing with multiple failure modes, considering mostly independent failure modes. In this paper, we present an optimal Bayesian control scheme for early fault detection of the gear system with dependent competing risks. The system failures include degradation failure and catastrophic failure. A three‐state continuous‐time–homogeneous hidden Markov model (HMM), namely the model with unobservable healthy and unhealthy states, and an observable failure state, describes the deterioration process of the gear system. The condition monitoring information as well as the age of the system are considered in the proposed optimal Bayesian maintenance policy. The objective is to maximize the long‐run expected average system availability per unit time. The maintenance optimization model is formulated and solved in a semi‐Markov decision process (SMDP) framework. The posterior probability that the system is in the warning state is used for the residual life estimation and Bayesian control chart development. The prediction results show that the mean residual lives obtained in this paper are much closer to the actual values than previously published results. A comparison with the Bayesian control chart based on the previously published HMM and the age‐based replacement policy is given to illustrate the superiority of the proposed approach. The results demonstrate that the Bayesian control scheme with two dependent failure modes can detect the gear fault earlier and improve the availability of the system.