While Cadmium Telluride (CdTe) excels in terms of photon radiation absorption properties and outperforms silicon (Si) in this respect, the crystal growth, characterization and processing into a radiation detector is much more complicated. Additionally, large concentrations of extended crystallographic defects, such as grain boundaries, twins, and tellurium (Te) inclusions, vary from crystal to crystal and can reduce the spectroscopic performance of the processed detector. A quality assessment of the material prior to the complex fabrication process is therefore crucial. To locate the Te-defects, we scan the crystals with infrared microscopy (IRM) in different layers, obtaining a 3D view of the defect distribution. This provides us with important information on the defect density and locations of Te inclusions, and thus a handle to assess the quality of the material. For the classification of defects in the large amount of IRM image data, a convolutional neural network is employed. From the post-processed and analysed IRM data, 3D defect maps of the CdTe crystals are created, which make different patterns of defect agglomerations inside the crystals visible. In total, more than 100 crystals were scanned with the current IRM setup. In this paper, we compare two crystal batches, each consisting of 12 samples. We find significant differences in the defect distributions of the crystals.