The thermal conductivity of graphene-reinforced aluminum-based composites was numerically investigated using the finite element method. Firstly, the influence of graphene content, morphology, and orientation on the thermal conductivity of the composite material was studied without interfaces. After obtaining the optimal parameters without interfaces, the effects of interface types and interface thickness were further investigated. The simulation results demonstrated that the thermal conductivity of the composite material improved with increasing graphene content and particle size. The thermal conductivity reached its maximum value when the orientation of the graphene was parallel to the direction of heat flow. When the thermal conductivity of the interface material was greater than that of the matrix, the thermal conductivity of the composite material increased more rapidly with increasing thickness. Conversely, when the thermal conductivity of the interface material was lower than that of the matrix, a smaller intrinsic thermal conductivity resulted in a faster decline in the thermal conductivity of the composite material with increasing thickness. This study reveals that the modification of graphene plays an important role in regulating the thermal conductivity of aluminum matrix composites.