Stripe noise removal continues to be an active field of research for remote image processing. Most existing approaches are prone to generating artifacts in extreme areas and removing the stripe-like details. In this paper, a weighted double sparsity unidirectional variation (WDSUV) model is constructed to reduce this phenomenon. The WDSUV takes advantage of both the spatial domain and the gradient domain's sparse property of stripe noise, and processes the heavy stripe area, extreme area and regular noise corrupted areas using different strategies. The proposed model consists of two variation terms and two sparsity terms that can well exploit the intrinsic properties of stripe noise. Then, the alternating direction method of multipliers (ADMM) optimal solver is employed to solve the optimization model in an alternating minimization scheme. Compared with the state-of-the-art approaches, the experimental results on both the synthetic and real remote sensing data demonstrate that the proposed model has a better destriping performance in terms of the preservation of small details, stripe noise estimation and in the mean time for artifacts' reduction.