This paper presents a novel method for registering panoramic images and 3D point clouds using the shape of the overall ground object in the scene as registration primitives. Firstly, a semantic segmentation method is applied to the panoramic image to extract the ground object and remove the sky. Next, the cloth simulation filtering algorithm (CSF) is employed to eliminate the ground points in the 3D point cloud. The remaining 3D ground objects are then projected onto a two-dimensional plane using the imaging model of the panoramic camera to obtain the registration primitives. Finally, we adopt the whale algorithm to perform a coarse-to-fine registration, utilizing overlap degree and mutual information as the similarity measures. The proposed method is evaluated in four different scenes and compared with the other four registration methods. The results demonstrate that the proposed method is accurate and effective, with an average registration error of 11.48 pixels (image resolution is 11000x5500 pixels) compared to the EOPs of the system of 101.67 pixels.