In this paper, the influence of aggregate sieving and distribution on the shielding performance of concrete against nuclear radiation is investigated numerically. A thermomechanical and neutron diffusion coupled numerical formulation is proposed to address the specific problem related to irradiated concrete. This model solves the multiphysics problem by combining a two-group neutron diffusion theory, with the thermal conduction equation adapted to the radiation condition, and the mechanical equilibrium problem. Concrete is modeled at the mesoscale as a two-phase material, comprising mortar matrix and aggregates. Three grading curves of Fuller type are simulated, different for the nominal diameter size intervals of the polyhedral aggregates, which are randomly distributed within cubic samples of concrete composite material. A boron-rich material, colemanite, is utilized as aggregate in the numerical simulations. The results show that the fast neutron flux is not much influenced by the variation of the grading curves. On the other hand, a slight variation of the coarse aggregate volume fraction is found to affect to some extent the thermal flux. As regards temperature and stress distribution, it is noted the barrier effects that the inclusions bring to the state variable diffusion process.