Malaria is a complex disease with many factors influencing the transmission dynamics, including age. This research analyzes the transmission dynamics of malaria by developing an age-structured mathematical model using the classical integer order and Atangana–Baleanu–Caputo fractional operators. The analysis of the model focused on several important aspects. The existence and uniqueness of solutions of fractional order were explored based on some fixed-point theorems,such as Banach and Krasnoselski. The Positivity and boundedness of the solutions were also investigated. Furthermore, through mathematical analysis techniques, we analyzed different types of stability results, and the results showed that the disease-free equilibrium point of the model is proved to be both locally and globally asymptotically stable if the basic reproduction number is less than one, whereas the endemic equilibrium point of the model is both locally and globally asymptotically stable if the basic reproduction number is greater than one. The findings from the sensitivity analysis revealed that the most sensitive parameters, essential for controlling or eliminating malaria are mosquito biting rate, density-dependent natural mortality rate, clinical recovery rate, and recruitment rate for mosquitoes. Numerical simulations are also performed to examine the behavior of the model for different values of the fractional-order alpha,and the result revealed that as the value α reduces from 1, the spread of the endemic grows slower. By incorporating these findings, this research helps to clarify the dynamics of malaria and provides information on how to create efficient control measures.