Digital micromirror device (DMD) based lithography system, which generates the mask pattern via a spatial light modulator, is increasingly applied in micro-nano fabrication due to its high flexibility and low cost. However, the exposure image is subject to distortion because of the optical proximity effect and the non-ideal system conditions. Correcting mask pattern with calibrated imaging model is an essential approach to improve the image fidelity of DMD-based lithography system. This paper introduces an imaging model calibration method for the DMD-based lithography testbed established by our group. The error convolution kernel and the point spread function in the imaging model are optimized using the batch gradient descent algorithm to fit a set of training data, which represent the impacts of non-ideal imaging process of the DMD-based lithography testbed. Based on the calibrated imaging model, the steepest descent algorithm is used to correct the mask pattern, thus improving the image fidelity of the testbed. Experiments demonstrate the effectiveness of the proposed model calibration method. It also shows that the size of error convolution kernel significantly influences the accuracy of the calibrated imaging model within a certain range. Finally, the effectiveness of the mask correction method is proved by experimental results.