The compaction quality is directly related to the deformation and stability of the rockfill dam. Measuring the test pit volume efficiently and accurately is the most critical step during the compaction quality inspection. A new method for calculating the test pit volume based on point cloud data is proposed. An auxiliary device that can change the scanning distance and angle of the handheld 3D laser scanner is developed to collect the initial point cloud. The segmentation method of the initial point cloud data including the test pit and the compaction surface outside the pit is to divide the data into two parts according to the order number of the segmentation point, after slicing and sorting point clouds, which is the key to ensuring the computational precision. The segmentation points are the adjacent two points with the greatest order number difference in these point clouds whose distance from the line connecting the end points of the slicing point clouds is less than dz. The compaction surface point clouds are used to construct a plane by the least-squares algorithm so that the closed three-dimensional model is formed by registering it with the test pit point clouds. After converting the test pit surface to the horizontal plane by the Rodrigues formula, the test pit point clouds are divided into n2 parts with equal projection areas on the horizontal plane, and n2 prisms are constructed using them and their projection areas. The test pit volume is the sum of the intersection space volumes of all prisms and the test pit model, and the intersection space is determined by comparing the Z-values of point clouds. The new method was programmed in MATLAB and applied to the Shuangjiangkou rockfill dam with a height of 315 m. The relative error of volume results between the new method and the old water-filling method is 0.14–2.31%. The cause of the error is analyzed, and it is proved that the results of the new method are closer to the real volume of the test pit in theory. This method reduces the inspection cost but greatly improves the level of precision, efficiency, and intelligence for compaction quality inspection.