Lensless digital holographic microscopy holds significant importance in areas such as environmental monitoring and biological specimen analysis. In order to address the challenges of cumbersome procedures and suboptimal recovery results when obtaining amplitude and phase information of objects in lensless imaging, in this paper, we propose a purely physics-supervised trained Fourier neural operator network (PSF-Net) for holographic particle imaging. Training the network solely using a few holographic particle images in the absence of ground truth. Once parameter optimization is complete, holographic reconstruction can be achieved without obtaining the defocus distance. A lensless holographic system is set up to capture holograms of particle fields. The proposed network is employed for amplitude and phase reconstruction, and its performance is compared with other methods. The results demonstrate that our proposed method exhibits superior performance in terms of reconstruction quality, noise resistance, and twin image elimination.