Optical phased array (OPA) imaging technique, which uses electro-optic modulation to achieve beam steering rather than mechanical scanning, is a raster scanning imaging method with great potential due to its noninertia and high speed. However, fabrication imperfection of an OPA causes pre-designed phase modulations not yielding desired steering angles, and a time-consuming calibration is usually required before practical use. Alternatively, it is possible to obtain images with a random phase modulation OPA. In this paper, we propose a fast calibration for the random phase modulation OPA LiDAR. Experimental results demonstrate that, to obtain images of the same quality, the proposed calibration is three times faster than the calibration used in raster scanning scheme. In the meantime, the proposed calibration simultaneously retrieves the point spread function of the imaging system during the process, which can be used to remove the image blurring caused by the sidelobes and further improve the image quality.