Optical axis jitters, or vibrations, which arise from wind shaking and structural oscillations of optical platforms, etc., cause a deleterious impact on the performance of adaptive optics systems. When conventional integrators are utilized to reject such high frequency and narrow-band disturbance, the benefits are quite small despite their acceptable capabilities to reject atmospheric turbulence. In our case, two suits of complete adaptive optics systems called double overlapped adaptive optics systems (DOAOS) are used to counteract both optical jitters and atmospheric turbulence. A novel algorithm aiming to remove vibrations is proposed by resorting to combine the Smith predictor and notch filer. With the help of loop shaping method, the algorithm will lead to an effective and stable controller, which makes the characteristics of error transfer function close to notch filters. On the basis of the spectral analysis of observed data, the peak frequency and bandwidth of vibrations can be identified in advance. Afterwards, the number of notch filters and their parameters will be determined using coordination descending method. The relationship between controller parameters and filtering features is discussed, and the robustness of the controller against varying parameters of the control object is investigated. Preliminary experiments are carried out to validate the proposed algorithms. The overall control performance of DOAOS is simulated. Results show that time delays are a limit of the performance, but the algorithm can be successfully implemented on our systems, which indicate that it has a great potential to reject jitters.