Accurate morphological evaluation via imaging of sperm cells is a critical step in both routine sperm analysis and sperm cell selection for in vitro fertilization. In this paper, we review the significant advantages of interferometric phase microscopy (IPM), also called digital holographic microscopy, for label-free sperm imaging in fertility clinics. We first review the current state of the art of label-free sperm cell imaging for in vitro fertilization, including brightfield, Zernike phase contrast, differential interference contrast (DIC), Hoffman modulation contrast and polarized light microscopy, as well as intracytoplasmic morphologically selected sperm injection (IMSI). Using experimental demonstrations, we compare the limitations of these techniques for label-free sperm imaging. IPM and its advantages are then discussed in detail. These include the ability to receive quantitative data regarding the 3D structure of sperm cells, as well as data concerning sperm biomass and morphology without labeling. The current gap to clinical use is discussed, as well as the possible solution of employing compact and simple holographic modules, which may allow holographic imaging to reach the clinic in the near future.