The dispersion and mixing of passive particles in a turbulent channel flow is investigated by means of a network-based representation of their motion. We employ direct numerical simulations at five different Reynolds numbers, from Reτ = 180 up to Reτ = 950, and obtain sets of particle trajectories via numerical integration. By dividing the channel domain into wall-normal levels, the motion of particles across these levels is used to build a time-varying complex network, which is able to capture the transient phase of the wall-normal mixing process and its dependence on the Reynolds number, Reτ . Using network metrics, we observe that the dispersion of clouds of tracers depends highly on both their wall-normal starting position and the time elapsed from their release. We identify two main mechanisms that contribute to the long lasting inhibition of the dispersion of particles released near the walls. We also show how the relative importance of these mechanisms varies with the Reynolds number. In particular, at low Reτ the weaker velocity fluctuations appear dominant in inhibiting dispersion, while at higher Reynolds numbers a larger role is played by cyclic patterns of motion. At the higher Reynolds numbers employed in this work, we find that most network properties are Reynolds-independent when scaled with outer flow variables. Instead, at lower Reτ , the aforementioned scaling is not observed. We explore the meaning of the emergence of this scaling in relation to the features of dispersion and to the network definition.