Direct converting quantum counting detectors based on cadmium telluride and cadmium zinc telluride have been investigated with respect to their properties under intense X-ray irradiation. To derive a detailed picture of the performance of such detectors, the influence of the electric field, the detector thickness, the temperature and the intensity of the X-ray irradiation was studied. The results are discussed in terms of the "polarization" phenomenon, a reduction of the electric field strength inside the detector due to immobile charge carriers accumulating during irradiation. Furthermore, the impact of Te-inclusions and -precipitates is presented.