Calibration is essential for the three-dimensional light field endoscope, and the two-step calibration method based on line features was proposed by us to accomplish it before. In the second step of the two-step calibration method, the relationship between the projections of virtual feature points on the microlens image and the central sub-aperture image was used to calibrate the parameters about the microlens array, but we find that the feasibility of the calibration method has a depth constraint on the border of the checkerboard. In this paper, we deduce and demonstrate this constraint, and an optimization algorithm is designed to improve the accuracy of the second step calibration due to the inaccurate black-and-white boundary detection. Experimental results show that our method is effective and accurate for the calibration of the three-dimensional light field endoscope.