External sulfate attack is an important factor causing a decrease in the mechanical properties of cement-based materials. In this paper, a computational prediction model of elastic modulus, considering the characteristics of sulfate corrosion from outside to inside and the influence of the interface transition zone (ITZ), was established to predict the elastic modulus of mortar under the external sulfate attack. Firstly, the backscattered electron (BSE) images of mortar and the algorithm of image threshold segmenting were used to determine a reasonable thickness of corroded ITZ. Secondly, the nanoindentation test was adopted to acquire the microscopic elastic parameters of phases (sand, cement, and ITZ) in corroded mortar. Moreover, the mortar mix proportion and Lu and Torquato’s model were adopted to calculate the volume fractions of phases. Finally, a computational prediction model of elastic modulus of mortar under sulfate attack was proposed with homogenization methods. The results indicate that the thickness of corroded ITZ is 20 mm, and the error values of elastic modulus between the theoretical prediction results and the experimental results are within 8%, indicating that the macroscopic elastic modulus of corroded mortar can be precisely predicted by the computational prediction model of elastic modulus.