This paper presents a novel three-phase damage model for the prediction of the post-peak responses of composite materials, such as recycled aggregate concrete (RAC). The proposed damage model is based on composite damage mechanics and is composed of three phases: cement paste, interface transition zone (ITZ), and aggregate. All phases are assumed to be linearly elastic and isotropic materials. The aggregate is supposed to be the undamaged material, while the cement paste and ITZ are considered the damaged materials. Three different composite damage models, namely Voigt (parallel), Reuss (serial), and the proposed generalized self-consistent (spherical), represent the damage growth in the composite materials. The Voigt parallel model is employed to address the upper bound of post-peak stiffness and stress, while the Reuss serial model represents the lower bound. To investigate the softening phenomenon after the post-peak state, both linear and exponential functions are used to describe the stress-strain curve in this state. Two numerical simulations are provided to examine the stress degradation in the softening state. Both simulations reveal that the post-peak stress degrades with increasing damage parameters and ITZ thickness. Therefore, both damage and ITZ’s thickness are significant factors for analyzing the post-peak responses of RAC.