In this paper, a novel approach is proposed to calibrate an underwater fisheye camera with the multi-frequency heterodyne technique based on the light field. First, an underwater fisheye camera imaging model based on the light field is established to demonstrate the multiple refractions of light in underwater propagation and the extraordinary imaging characteristics of the fisheye lens, which shows better conciseness and organization. Then, the calibration method applied in the underwater fisheye camera imaging system is proposed to determine the mapping of the light field and the pose of the underwater fisheye camera. The light field mapping of the fisheye lens is obtained with the assistance of the multi-frequency heterodyne technique. Finally, an underwater fisheye camera is developed and the experimental results are obtained to validate the correctness and feasibility of the proposed method. Our underwater imaging system proves capable of achieving higher accuracy in the context of underwater 3D measurement.