In this article, thermal performance of different waste materials and by-products of industrial processes is investigated experimentally. A geopolymer concrete block with 7.5 cm thickness and cross-sectional area of 5 × 5 cm was considered as a reference model to measure heat transmission across the two opposite surfaces while all four remnant surfaces were perfectly insulated. For all other samples, a sandwich concrete block was developed by taking two pieces of the geopolymer concrete with 2.5 cm thickness each on either side and insulation material of 2.5 cm thickness in between. The sandwich materials investigated were air cavity, expanded polystyrene foam, polyurethane foam, rubber tire, date palm, PCM-30, and PCM-42. Experimental investigations revealed that the investigated green materials and industrial by-products have comparable insulation performance with respect to the traditional insulations such as expanded polystyrene foam. It is found that polyurethane foam and date palm can reduce indoor cooling demand by 46.6% each in hot conditions while rubber tire can reduce indoor heating demand by 59.2% in cold climatic conditions at the maximum. The research results confirm and encourage the effective utilization of waste materials in building walls for reducing indoor air-conditioning demand in the extreme climatic conditions.