PurposeTo detect contrast media injection phase from CT images by means of organ segmentation and deep learning.Materials and MethodsA total number of 2509 CT images split into four subsets of non-contrast (class #0), arterial (class #1), venous (class #2), and delayed (class #3) after contrast media injection were collected from two CT scanners. Seven organs including the liver, spleen, heart, kidneys, lungs, urinary bladder, and aorta along with body contour masks were generated by pre-trained deep learning algorithms. Subsequently, five first-order statistical features including average, standard deviation, 10, 50, and 90 percentiles extracted from the above-mentioned masks were fed to machine learning models after feature selection and reduction to classify the CT images in one of four above mentioned classes. A ten-fold data split strategy was followed. The performance of our methodology was evaluated in terms of classification accuracy metrics.ResultsThe best performance was achieved by Boruta feature selection and RF model with average area under the curve of more than 0.999 and accuracy of 0.9936 averaged over four classes and ten folds. Boruta feature selection selected all predictor features. The lowest classification was observed for class #2 (0.9888), which is already an excellent result. In the ten-fold strategy, only 33 cases from 2509 cases (∼1.4%) were misclassified.ConclusionWe developed a fast, accurate, reliable, and explainable methodology to classify contrast media phases which may be useful in data curation and annotation in big online datasets or local datasets with non-standard or no series description.Key pointsThe lack of standard series description and information about contrast media phase limits the usability of medical CT data.We developed a twostep deep learning/machine learning solution with excellent performance.This fast, automated, reliable and explainable purposed pipeline can tag every CT images with using only image matrices.