Currently, the standard length measurement method with nanometre resolution is laser interferometry. However, it is difficult to determine an arbitrary length with an accuracy of sub-nanometre or less order using interferometers because they have a nonlinearity problem in fringe interpolation. A phase modulation homodyne interferometer (PMHI) that can be used to determine the optical path difference of an integer multiple of the wavelength (n × λ) with picometre resolution was proposed by IMGC (former Italian Standard Institute). The lattice spacing of approximately 0.246 nm for graphite regular crystalline lattices is uniform and stable over a long range, when the crystals are stress free. These crystals can be used as a reference scale with sub-nanometre resolution. The scanning tunnelling microscope (STM) is emerging as a powerful tool in surface engineering, and enables us to image atoms on a crystalline surface. Therefore, such a crystalline surface can be used as ‘the crystalline scale’ using the STM.In this study, an instrument for calibrating optical encoders is developed by combining a graphite crystalline lattice as a fine scale and the optical fringe of the PMHI as a coarse scale. The instrument consists of a precise linear X-axis sample stage, on which the reference graphite crystal and the optical encoder scale are set, a head of the STM with a YZ tip scanner and a PMHI. The relative displacement of the X-axis sample stage between optical interference dark fringes (= null points) of the PMHI, which is λ/16 times the integer value in the design, can be measured with picometre resolution using the phase modulation technique. A lattice spacing of 0.246 nm on the graphite crystalline surface is derived as the fine scale from the STM image and the optical fringes of the PMHI. In the experiment, the periodical error of the optical encoder, whose minimum resolution is less than a nanometre, is measured using both the lattice spacing of graphite and the optical fringes of the PMHI. The results show that the proposed instrument has the feasibility to calibrate optical encoders with an uncertainty of 10 pm order.