The origin of the effect of inhomogeneous swelling observed near grain boundaries in irradiated materials is examined taking into account both nucleation and diffusional growth of cavities, and the interaction of cavities with mobile interstitial clusters produced in collision cascades. The model shows the formation of a characteristic profile of inhomogeneous swelling that exhibits features similar to those observed experimentally. The rate of swelling is found to be strongly dependent on the size of cavities, with cavities growing near the boundary being able to reach substantially larger sizes than those growing in the interior area of the grain. The distance z* between the peak of swelling and the grain boundary scales with the density of cavities N 0 as z*ϳN 0 Ϫ , where  is close to 1/3.