Nilotinib hydrochloride monohydrate (NHM) is an anti-cancer drug whose solubility was statically determined in supercritical carbon dioxide (SC-CO2) for the first time at various temperatures (308–338 K) and pressures (120–270 bar). The mole fraction of the drug dissolved in SC-CO2 ranged from 0.1 × 10–5 to 0.59 × 10–5, corresponding to the solubility range of 0.016–0.094 g/L. Four sets of models were employed to evaluate the correlation of experimental data; (1) ten empirical and semi-empirical models with three to six adjustable parameters, such as Chrastil, Bartle, Sparks, Sodeifian, Mendez-Santiago and Teja (MST), Bian, Jouyban, Garlapati-Madras, Gordillo, and Jafari-Nejad; (2) Peng-Robinson equation of state (Van der Waals mixing rule, had an AARD% of 10.73); (3) expanded liquid theory (modified Wilson model, on average, the AARD of this model was 11.28%); and (4) machine learning (ML) algorithms (random forest, decision trees, multilayer perceptron, and deep neural network with respective R2 values of 0.9933, 0.9799, 0.9724 and 0.9701). All the models showed an acceptable agreement with the experimental data, among them, the Bian model exhibited excellent performance with an AARD% of 8.11. Finally, the vaporization (73.49 kJ/mol) and solvation (− 21.14 kJ/mol) enthalpies were also calculated for the first time.