The radiation resistance of CaF2 crystal is one of the critical properties in the application of deep ultraviolet lithography, but the damage process of CaF2 crystal under 193 nm laser irradiation is still unclear. This paper reports the damage behavior of CaF2 crystals under 193 nm laser irradiation and the key defect factors affecting the damage. Through the 193 nm laser irradiation experiment, it is found that the crystal damage is mainly manifested as the radiation-induced color centers inside the crystal and the radiation-induced damage pits on the surface. Irradiation-induced color centers were analyzed by UV-visible spectrophotometer, and linear fitting was performed between absorption coefficients of different color centers and Y impurity contents. The results show that the Y ion has a low-order orbit that overlaps with the F center structure wave function, and hybridizes to form a stable structure. Linear fitting results show that there is a linear relationship between Y ions contents and the intrinsic color centers of CaF2 crystals, confirming that Y element is the key impurity ion affecting the formation of color centers. Energy dispersive X-ray spectrometer (EDS) and electron backscatter diffraction (EBSD) were used to characterize the elements distribution and structure defects of radiation-induced damage pits. EDS results show that the content of calcium in the damage pits increases and the content of fluorine decreases, which confirms that the diffusion of H centers and the aggregation of F centers lead to irradiation damage. EBSD results show that surface irradiation damage occurs preferentially at dislocations. Therefore, reducing the impurity content and dislocation density is an important way to improve the anti-irradiation damage performance of calcium fluoride crystals under 193 nm laser.