Remote sensing images have been used in many fields, such as urban planning, military, and environment monitoring, but corruption by stripe noise limits its subsequent applications. Most existing stripe noise removal (destriping) methods aim to directly estimate the clear images from the stripe images without considering the intrinsic properties of stripe noise, which causes the image structure destroyed. In this paper, we propose a new destriping method from the perspective of image decomposition, which takes the intrinsic properties of stripe noise and image characteristics into full consideration. The proposed method integrates the unidirectional total variation (TV) regularization, group sparsity regularization, and TV regularization together in an image decomposition framework. The first two terms are utilized to exploit the stripe noise properties by implementing statistical analysis, and the TV regularization is adopted to explore the spatial piecewise smooth structure of stripe-free image. Moreover, an efficient alternating minimization scheme is designed to solve the proposed model. Extensive experiments on simulated and real data demonstrate that our method outperforms several existing state-of-the-art destriping methods in terms of both quantitative and qualitative assessments.