Chloride diffusivity is the most crucial factor for evaluating durability and predicting service life of concrete structures exposed to chloride environment. In the present paper, concrete is considered as a random heterogeneous composite of three phases: the aggregates, the matrix, and the interfacial transition zones (ITZ) between them. A mesoscopic model has been established based on the random aggregate model and a numerical model for calculating the chloride diffusivity has been proposed. The influences of aggregate size and the properties of the interfacial transition zones on chloride diffusivity of concrete are also discussed. The chloride diffusivity is size dependent because aggregates are randomly distributed in concrete. The chloride diffusivity under different sizes is analyzed by numerous numerical simulations, and the representative volume element (RVE) of chloride penetration into concrete is researched by the statistical method.