The cement industry generates very large amounts of CO2 into the atmosphere. In recent years, there has been a search for alternative cementitious materials and micro-fillers that could partially or fully replace cement in cement composites without compromising their durability. This paper investigates the possibility of using brick powder (BP) and clay powder (CP) as a partial replacement for cement (up to 20% by weight) in cement paste. The raw materials were characterized, and the physical and mechanical properties of the modified cement pastes were studied, as well as their resistance to a short-term thermal shock at 250 °C. The study was supplemented by intelligent modelling of compressive strength using the support vector machine (SVM) algorithms. The results indicated a significant increase in tensile strength (up to 100%) and an increase in thermal resistance of cement pastes modified with BP and CP. The proposed SVM model had high accuracy (R2 = 0.90), indicating its suitability to predict the compressive strength of the modified cement matrix. This study complements the knowledge in the field of inter alia, the effect of a short-term thermal shock at elevated temperature on the properties of BP and CP modified cement paste, and the effect of BP, which, due to its grain size, plays more the role of a microfiller than a pozzolanic additive.