The demolition of buildings, apart from being energy intensive and disruptive, inevitably produces construction and demolition waste (C&Dw). Unfortunately, even today, the majority of this waste ends up underexploited and not considered as valuable resources to be re-circulated into a closed/open loop process under the umbrella of circular economy (CE). Considering the amount of virgin aggregates needed in civil engineering applications, C&Dw can act as sustainable catalyst towards the preservation of natural resources and the shift towards a CE. This study completes current research by presenting a life cycle inventory compilation and life cycle assessment case study of two buildings in France. The quantification of the end-of-life environmental impacts of the two buildings and subsequently the environmental impacts of recycled aggregates production from C&Dw was realized using the framework of life cycle assessment (LCA). The results indicate that the transport of waste, its treatment, and especially asbestos’ treatment are the most impactful phases. For example, in the case study of the first building, transport and treatment of waste reached 35% of the total impact for global warming. Careful, proactive, and strategic treatment, geolocation, and transport planning is recommended for the involved stakeholders and decision makers in order to ensure minimal sustainability implications during the implementation of CE approaches for C&Dw.