Oil tea (Camellia oleifera) is one of the world’s major woody edible oil plants and is vital in providing food and raw materials and ensuring water conservation. The yield of oil tea can directly reflect the growth condition of oil tea forests, and rapid and accurate yield measurement is directly beneficial to efficient oil tea forest management. Light detection and ranging (LiDAR), which can penetrate the canopy to acquire the geometric attributes of targets, has become an effective and popular method of yield identification for agricultural products. However, the common geometric attribute information obtained by LiDAR systems is always limited in terms of the accuracy of yield identification. In this study, to improve yield identification efficiency and accuracy, the red-green-blue (RGB) and luminance-bandwidth-chrominance (i.e., YUV color spaces) were used to identify the point clouds of oil tea fruits. An optimized mean shift clustering algorithm was constructed for oil tea fruit point cloud extraction and product identification. The point cloud data of oil tea trees were obtained using terrestrial laser scanning (TLS), and field measurements were conducted in Changsha County, central China. In addition, the common mean shift, density-based spatial clustering of applications with noise (DBSCAN), and maximum–minimum distance clustering were established for comparison and validation. The results showed that the optimized mean shift clustering algorithm achieved the best identification in both the RGB and YUV color spaces, with detection ratios that were 9.02%, 54.53%, and 3.91% and 7.05%, 62.35%, and 10.78% higher than those of the common mean shift clustering, DBSCAN clustering, and maximum-minimum distance clustering algorithms, respectively. In addition, the improved mean shift clustering algorithm achieved a higher recognition rate in the YUV color space, with an average detection rate of 81.73%, which was 2.4% higher than the average detection rate in the RGB color space. Therefore, this method can perform efficient yield identification of oil tea and provide a new reference for agricultural product management.