Since the surface of the target which the self-mixing interferometric displacement sensor encounters generally is rough, the sensor is always affected by speckle effect. Although the speckle has an effect on the quality of the self-mixing signal, we have found for the first time that the speckle puts forward an important requirement for the optical system of the sensor. In the paper we have also presented a design method of the optical system for the self-mixing displacement sensor which can provide variable measurement distance. The methods and results reported will give deeper understanding of self-mixing phenomenon and be of great help to improving the performance of the self-mixing sensor.