This paper is about geometric calibration of the high resolution CT (Computed Tomography) system. Geometric calibration refers to the estimation of a set of parameters that describe the geometry of the CT system. Such parameters are so important that a little error of them will degrade the reconstruction images seriously, so more accurate geometric parameters are needed in the higher-resolution CT systems. But conventional calibration methods are not accurate enough for the current high resolution CT system whose resolution can reach sub-micrometer or even tens of nanometers. In this paper, we propose a new calibration method which has higher accuracy and it is based on the optimization theory. The superiority of this method is that we build a new cost function which sets up a relationship between the geometrical parameters and the binary reconstruction image of a thin wire. When the geometrical parameters are accurate, the cost function reaches its maximum value. In the experiment, we scanned a thin wire as the calibration data and a thin bamboo stick as the validation data to verify the correctness of the proposed method. Comparing with the image reconstructed with the geometric parameters calculated by using the conventional calibration method, the image reconstructed with the parameters calculated by our method has less geometric artifacts, so it can verify that our method can get more accurate geometric calibration parameters. Although we calculated only one geometric parameter in this paper, the geometric artifacts are still eliminated significantly. And this method can be easily generalized to all the geometrical parameters calibration in fan-beam or cone-beam CT systems.