Over the past decade a considerable number of studies have focused on generating semantically rich as-built building information models (BIMs). However, the prevailing methods rely on laborious manual segmentation or automatic but error-prone segmentation. In addition, the methods failed to make good use of existing semantics sources. This article presents a novel segmentation-free derivative-free optimization (DFO) approach that translates the generation of asbuilt BIMs from 2D images into an optimization problem of fitting BIM components regarding architectural and topological constraints. The semantics of the BIMs are subsequently enriched by linking the fitted components with existing semantics sources. The approach was prototyped in two experiments using an outdoor and an indoor case, respectively. The results showed that in the outdoor case 12 out of 13 BIM components were correctly generated within 1.5 hours, and in the indoor case all target BIM components were correctly generated with a root-mean-square deviation (RMSD) of 3.9 cm in about 2.5 hours. The main computational novelties of this study are: (1) to translate the automatic as-built BIM generation from 2D images as an optimization problem; (2) to develop an effective and segmentation-free approach that is fundamentally different from prevailing methods; and (3) to exploit online open BIM component information for semantic enrichment, which, to a certain extent, alleviates the dilemma between information inadequacy and information overload in BIM development.