Shape segmentation in urban environments forms the foundation for tasks such as classification and reconstruction. Most artificial buildings with complex structures are composed of multiple simple geometric primitives. Based on this assumption, this paper proposes a divisive hierarchical clustering algorithm that uses shape classification and outliers reassignment to segment LiDAR point clouds in order to effectively identify the various shapes of structures that make up buildings. The proposed method adopts a coarse-to-fine strategy. Firstly, based on the geometric properties of different primitives in a Gaussian sphere space, coarse extraction is performed using Gaussian mapping and the DBSCAN algorithm to identify the primary structure of various shapes. Then, the error functions are constructed after parameterizing the recognized shapes. Finally, a minimum energy loss function is built by combining the error functions and binary integer programming (BIP) to redistribute the outlier points. Thereby, the accurate extraction of geometric primitives is achieved. Experimental evaluations on real point cloud datasets show that the indicators of precision, accuracy, and F1 score of our method are 0.98, 0.95, and 0.96 (point assignment) and 0.97, 0.95, and 0.95 (shape recognition), respectively. Compared with other state-of-the-art methods, the proposed method can efficiently segment planar and non-planar structures with higher quality from building point clouds.