The vibration-based analysis is an effective technique for planetary gearbox fault diagnosis, but a difficult task is how to accurately identify fault features from noisy vibration signals. In this paper, a nonconvex wavelet thresholding total variation (WATV) denoising method is proposed for planetary gearbox fault diagnosis, which combines wavelet-domain sparsity and total variation (TV) regularization. The TV regularization algorithm is employed to modify the retained wavelet coefficients so that the occurrence of oscillations caused by wavelet thresholding is suppressed. To overcome the underestimation shortcoming of L1-norm regularization, nonconvex penalty function regularization is used to strongly promote the sparsity of estimation while guaranteeing that the global optimal solutions are obtained even though the objective function is nonconvex. Then, the split augmented Lagrangian shrinkage (SALSA) method is developed to solve the nonconvex WATV denoising problem. Two experimental studies are performed to verify the performance and effectiveness of the proposed method. Comparisons with the soft thresholding and basis pursuit denoising (BPD) methods show that the proposed method can accurately estimate the fault features from vibration signals, which means that the proposed method is an effective and promising tool for planetary gearbox fault diagnosis.