Due to the low impedance matching caused by the high dielectric permittivity of graphene, the strong absorption of electromagnetic waves by graphene/polymer nanocomposites is challenging. In this paper, an analytical model for microwave absorption based on Maxwell’s equation and the effective medium theory, considering the interface effect, was constructed to explore the effect of the gradient distribution of graphene in the polymer matrix on its microwave absorption performance. The outcome indicated that the impedance of the composites matched well with the air, and its attenuation ability for electromagnetic waves was obviously improved as the graphene concentration was distributed in a gradient form. For instance, when the thickness of the material is 10 mm, based on the optimal concentration of the homogeneous composites being 0.7 wt%, the graphene concentration range of the gradient composites is set to 0.7–0.9 wt% and distributed in three gradient forms of linear, parabolic, and 0.5 power. The results show that the microwave absorption performance is significantly improved compared with the homogeneous composites. Among them, the effective bandwidth on the 0.5 power distribution is 5.2 GHz, 0.5 GHz higher than that of the homogeneous composites. The minimum reflection loss (RL) is as low as −54.7 dB, which is 26.26 dB lower than that of the homogeneous composites. This paper contributes to the design and application of gradient absorbing structures.