Mathematical modeling serves as a powerful tool for investigating the dynamics of diseases and the efficacy of various treatment strategies. This study proposes a fractional-order mathematical model for lung cancer progression, formulated using the Caputo fractional derivative. The model categorizes the biological system into four distinct cell classes: tumor cells (T), active macrophage cells (A), macrophage cells (M), and normal tissue cells (N). By leveraging operational matrices, the complex system of differential equations
is transformed into a system of algebraic equations. To solve this nonlinear system, the norm-2 of residual functions is minimized, converting the problem into an optimization challenge. To address this, we employ generalized Bessel polynomials combined with the Lagrange multipliers method. Our proposed hybrid approach improves upon existing techniques such as the traditional Bessel polynomials and Lagrange polynomial methods by offering enhanced computational efficiency and higher accuracy in identifying optimal solutions. The efficacy of the model is demonstrated through numerical simulations, which show its robustness in capturing the intricate behaviors of the cancer-tissue interaction. These features make our proposed method a promising tool for the development of advanced biomedical models, with potential applications in optimizing treatment protocols for complex diseases like lung cancer.