Weld stability is directly related to the safety and reliability of engineering, and continuous improvement of its detection technology has great research significance. This paper presents a novel method for weld stability detection based on color digital holography. A color digital holography optical path was designed to capture the holograms of welds under varying loads. Several common denoising algorithms were used to deal with speckle noise in the wrapped phase, among which the 4-f optical simulation integrated cycle-consistent generative adversarial network (4f-CycleGAN) denoising algorithm based on deep learning was more suitable for the color digital holographic detection system. The three-dimensional deformation fields of three samples (lap-jointed, butt-jointed, and defective butt-jointed aluminum alloy plates) under different loads were calculated. The center profile of the deformation field in the direction of load and holographic reconstruction images are used to determine the position of the weld. The coefficient of variation near the weld was used to evaluate the weld stability. The coefficient of variation for lap-jointed, butt-jointed and defective butt-jointed plates are 0.0335 (<0.36, good stability), 0.1240 (<0.36, good stability) and 0.3965 (>0.36, poor stability), respectively. The research results of this paper provide a new strategy for detecting weld stability.