Although straw is being utilized as an additive in construction materials, the thermal properties of straw and building materials in combination are insufficient. The thermal properties of straw–mortar composite materials can be improved by the pretreatment of straw. The alkali treatment enhances the mechanical attachment between the fibers and the matrix material, assuring that the straw–mortar composite materials have solid thermal insulation characteristics. Pretreatment with sodium hydroxide was utilized in this work to enhance the thermal properties of straw–mortar composite materials. This study mainly investigated the thermal properties of straw–mortar composite material after sodium hydroxide pretreatment and its change rules under the condition of the freeze–thaw cycle. A three-factor, three-level Box–Behnken experimental design, with the straw content (%), pretreatment time (min), and reagent concentration (%) as process parameters, was used. The response variables were the thermal conductivity, thermal diffusivity, and thermal resistance. The findings revealed that all of the variables had a substantial impact on the replies. Optimization parameters of 17.95% for the straw content, 19.50 min for the pretreatment time, and 4.99% for the reagent concentration for the straw–mortar composite materials were achieved. A thermal conductivity of 0.211 W·(m·K)−1, a thermal diffusivity of 0.277 mm2·s−1, and a thermal resistance of 57.211 K·W−1 were the optimal thermal property indices. Furthermore, during the freeze–thaw cycle, the thermal conductivity coefficient and thermal diffusion coefficient of the combined pretreatment composite were 26% and 9% lower than the materials without the treatment. The thermal performance of the mortar composites prepared by alkali-treated straw was better than that prepared by untreated straw.