Building Information Modelling (BIM) has changed the way in which buildings are conceived, planned and executed. Apart from their frequent use for as-planned buildings, BIM authoring tools have now been adopted for a number of years for digitising existing buildings as well, mostly by performing a 'scan-to-BIM' process: the creation of a BIM model, primarily based on point clouds. However, some inherent characteristics of existing buildings are complicating such a process: uncertainties, geometric irregularities, classification of heritage building components, linking sources about the real-world asset and an interdisciplinarity that may go beyond traditional Architecture, Engineering and Construction (AEC) topics (e.g. heritage, Facility Management, sensor data and damage assessment). In this paper, a framework called 'scan-to-graph' (STG) is proposed to integrate the concepts of scan-to-BIM with Semantic Web technologies, as these could provide answers to the above-mentioned challenges, most notably on documentation of uncertainties, sources and modelling decisions, element classification and cross-discipline information linking. In order to test the STG concept, a use case was developed where the Audience Room of the Gravensteen castle in Ghent was reconstructed from point clouds, semantically enriched and stored as an RDF graph. The resulting graph contains multiple interlinked geometry types, metadata about the reconstruction process and the sources and allows to unambiguously refer to other contextual data on the Web.