To evaluate the radiation-induced degradation of concrete, a rigid-body spring network model is introduced that takes into account the three phases in concrete: mortar, aggregate, and the interfacial transition zone. The proposed model enables evaluation of the change in the physical properties of concrete affected by aggregate expansion under the free restraint condition. Good agreement with previous experimental data is found for the linear expansion of the concrete specimen and the compressive strength, Young's modulus, and splitting tensile strength. Based on the numerical results, it is concluded that, to reproduce the physical property changes in concrete, the expansion of mortar due to the radiation-induced expansion of fine aggregate and/or creep behavior must be considered. In addition, it is clarified that an isolated expansion of mortar with a lack of expansion in the coarse aggregate also degrades the concrete and, consequently, analysis of the type of aggregate used is critical for predicting the properties of concrete under neutron irradiation. Furthermore, the impact of inhomogeneous expansion of rock-forming minerals in coarse aggregates on physical property changes is studied, showing that such a partial expansion in the aggregates and the resultant cracks in aggregates greatly influences the reduction of the Young's modulus, with minimal impact on the reduction of compressive strength. The proposed model can be used to evaluate concrete degradation due to radiation-induced volumetric expansion of aggregate caused by the metamictization of rock-forming minerals.