Optical systems with high tolerance sensitivity can be significantly affected by errors in manufacturing and assembly. By discussing the change in the ray position on the image surface caused by the system tolerance, we propose a scheme by controlling the change of the chief ray on the image surface as little as possible with the chief ray drift, which is called the stability of the chief ray, to reduce the tolerance sensitivity. The effectiveness of the method is demonstrated by a design example. Since it is based on controlling the real rays of the optical system, this method can theoretically be applied to all optical systems design.