A rhodium-emitted X-ray with a photon energy at 20 KeV was employed to irradiate CaF2 with different surface finishing. Color centers associated with the irradiation damage were determined by spectral measurements. The depth distribution of the color centers was calculated and confirmed in a wedge spot generated by magnetorheological finishing. The results suggest that the color center formation is directly associated with the damage resistance of the optics. Laser durable grade CaF2 demonstrates higher damage resistance to the X-ray irradiation when compared to excimer ArF laser grade CaF2. Subsurface-damage-free CaF2 surfaces further enhance the damage resistance.