Cement-based material encapsulation is a method of encapsulating electronic devices in highly thermally conductive cement-based materials to improve the heat dissipation performance of electronic components. In the field of construction, a thermoelectric generator (TEG) encapsulated with cement-based materials used in the building envelope has significant potential for waste heat energy recovery. The purpose of this study was to investigate the effect of cement-based materials integrated with aluminum heatsinks on the heat dissipation of the TEG composite structure. In this work, three types of thermoelectric work units encapsulated with cement paste were proposed. Moreover, we explored the effect of encapsulated structure, heat dissipation area, the height of thermoelectric single leg, and heat input temperature on maintaining the temperature difference between the two sides of the thermoelectric single leg with COMSOL Multiphysics. The numerical simulation results showed that under the conditions of a heat source temperature of 313.15 K and ambient temperature of 298.15 K, the temperature difference between the two sides of the internal thermoelectric single leg of Type-III can maintain a stable temperature difference of 7.77 K, which is 32.14% higher than that of Type-I and Type-II (5.88 K), and increased by 26.82% in the actual experiment. This work provides a reference for the selection and application of TEG composite structures of cement-based materials combined with aluminum heatsinks.