In order to accurately and quickly predict the failure probability of gears with multiple failure modes, a novel reliability analysis methodology based on the mixed Copula (MCopula) function model is proposed to deal with the complex correlation among different failure functions. Firstly, we construct a novel MCopula model based on three famous Copula functions: Gumbel Copula, Clayton Copula, and Frank Copula functions. Secondly, we use and improve the particle swarm optimization (PSO) method to optimally calculate the weight coefficients in the proposed MCopula model. Thirdly, the maximum likelihood estimation (MLE) method is adopted to estimate related parameters in the proposed MCopula model. Finally, we verify the proposed reliability analysis methodology with a standard life-prediction case of a strut system and a practical life-problem case of a gear pair system. Comparison results of both cases show that, by using the proposed methodology, the failure probability of a gear pair system with multiple failure correlations can be quickly calculated through a small number of samples and can be estimated as accurately as that by the Monte Carlo scheme. Consequently, our proposed novel methodology successfully analyzes the reliability problems for a gear pair system with multiple failure modes. The proposed methodology can be further applied to solve the reliability problem for other machine parts.