Damage and failure of rubberized self-compacting concrete (RSCC) are studied by mesostructural models. The models include six phases: mortar, aggregates, rubber particles, aggregate-mortar interfacial transaction zones (A-M ITZs), rubber-mortar interfacial transaction zones (R-M ITZs), and voids. Thin layers between mortars and aggregates and between mortars and rubber particles represent A-M ITZs and R-M ITZs, , respectively. Aggregates and rubber particles are modeled with linear elastic, while mortars, A-M ITZs, and R-M ITZs are with different damage-plasticity behaviors. The mesoscale models are validated by the comparison of numerical results and experimental results. The effects of essential phase parameters on the composite’s strength are evaluated, and empirical laws for these effects are established by data regression. It is demonstrated that the effect of porosity, size, and content of rubber particles affect strength and toughness, which provides guidance to the design of such composites for practical applications.