In this study, we investigate a new fractional-order mathematical model which considers population dynamics among tumor cells-macrophage cells-active macrophage cells, and host cells involving the Caputo fractional derivative. Firstly, the stability of the positive steady state of the model is studied. Subsequently, the conditions for existence and uniqueness of the solutions are examined. Then, the least squares curve fitting method (LSCFM) which is one of the prominent methods for parameter estimation is used to fit the parameters of the model. It is aimed to fit the relevant parameters with the help of the tumor tissue samples which were collected from the patient with non-small cell lung cancer who had chemotherapy-naive hospitalized at Kayseri Erciyes University hospital in Turkey. A total of 12 parameters in the model are estimated using the data of lung tumor cells of this patient for 14 days. Moreover, the numerical simulations are given by considering the different fractional orders and different parameters for the model. So, it is achieved how the change in $$\alpha $$
α
affects the dynamic behavior of the system. In the sequel, to point out the advantages of the fractional-order modeling, the memory trace and hereditary traits are taken into consideration. Finally, the interpretations in terms of biological science are provided in conclusion. We believe that this interdisciplinary study will open new doors for other similar studies and will shed light on the studies to be developed on the use of real data in the mathematical modeling of cancer.