Considering the fact that the remote sensing image is mainly captured by a linear CCD with the push-broom way which the image varies over time, the time-varying wavelet packets for image restoration is proposed in the paper. On one hand, the result of the method is that the problem of the correlation between the images is solved, and on the other hand, it is the method that can remarkably reduce the calculating overhead and the data throughput, which is a key innovation for a real-time system. In this paper, the optimized wavelet packet bases by double tree searching algorithm are adaptively changed in different time. To realize the algorithm, we presented a dual DSPs real-time parallel system.The parallel system based on TMS320C6416-7E3 DSP has the characteristics of modular and flexible design and maintainability. The ping-pong structure and the streamline structure are both designed in the system. According to the complexity of the algorithm and the requirement of the data throughput, one of the two parallel structures can be realized freely only by changing a bit. It can realize the restoration algorithm with 4096*4096 images in real-time by demonstrated by our experiment in practice.