This paper examines the possibility of using crushed brick and recycled concrete as a partial replacement of natural aggregates in a production of prefabricated wall panels used for constructing nearly zero-energy buildings.Replacement ratios of natural aggregates by 40%, 50% and 60% were investigated. Mechanical, durability and thermal properties of recycled aggregate concrete were tested. The test results indicate the possibility of manufacturing concrete with recycled concrete aggregates and recycled brick aggregates, which can find wide use in the building construction process. In addition, energy efficiency and sustainability parameters of recycled aggregate concrete can be coupled with an exigent need to improve energy performance of the building stock in the European Union and neighbouring countries. It is shown in this paper that around 46% of embodied energy and around 39% of embodied carbon per panel can be saved for a life span of 50 years, compared to conventional structural concrete insulated panels.