Colloidal silica-bonded castables offer several advantages compared to traditional calcium aluminate cement (CAC)-bonded castables, including lower torque values during mixing, superior drying properties, and a lower CaO content. Nevertheless, information on the combination of CAC and colloidal silica is limited, and the effect of CAC additions on the drying properties of colloidal silica-bonded castables remains unknown. In this study, these drying properties were measured by rapidly heating 400 kg samples to 500 °C and assessing the resulting damage to each sample. Additionally, the physical and chemical properties of small-scale samples were analyzed to evaluate the impact of CAC addition. The analyzed properties included cold crushing strength (CCS), density, permanent linear change (PLC) and weight loss. The microstructure of the samples was investigated by FESEM and EDS. The results indicate that adding 1.5 wt.% increased the cold crushing strength at 20 °C, while lower CAC amounts had no noticeable effect. A mullite phase was observed in the sample without CAC, and correspondingly, anorthite was found in those with CAC additions. The samples exhibited significant differences in the drying tests, with the degrees of damage increasing with the CAC addition.