In this work, a nonequilibrium molecular dynamics simulation is utilized to explore the effect of network structure of graphene (GE) on the thermal conductivity of the GE/polydimethylsiloxane (PDMS) composite. First, the thermal conductivity of composites rises with increasing volume fraction of GE. The heat transfer ability via the GE channel is found to be nearly the same by analyzing the GE−GE interfacial thermal resistance (ITR). More heat energy is transferred via the GE channel at the high volume fraction of GE by calculating the GE heat transfer ratio, which leads to the high thermal conductivity. Then, the thermal conductivity of composites rises with increasing stacking area between GE, which is attributed to both the strong heat transfer ability via the GE channel and the high GE heat transfer ratio. Following it, the thermal conductivity of composites first rises and then drops down with increasing defect density for a single vacancy defect while it continuously increases for a single void defect. The heat transfer ability between GE is enhanced due to the formation of interlayer covalent bonds. However, the intrinsic thermal conductivity of GE is significantly reduced for a single vacancy defect while it remains relatively well for a single void defect. As a result, the GE heat transfer ratio is maximum at the intermediate defect density for a single vacancy defect while it rises monotonically for a single void defect, which can rationalize the thermal conductivity. Meanwhile, the relationship between ITR and the number of covalent bonds can be described by an empirical equation. Finally, the thermal conductivity for the stacked structure is larger than that for the noncontact structure or the intersected structure. In summary, this work provides a clear and novel understanding of how the network structure of GE influences the thermal conductivity of the GE/PDMS composite.