Filtering out vegetation from a point cloud based on color is only rarely used, largely due to the lack of knowledge of the suitability of input information (color, vegetation indices) and the thresholding methods. We have evaluated multiple vegetation indices (ExG, ExR, ExB, ExGr, GRVI, MGRVI, RGBVI, IKAW, VARI, CIVE, GLI, and VEG) and combined them with 10 methods of threshold determination based on training set selection (including machine learning methods) and the renowned Otsu’s method. All these combinations were applied to four clouds representing vegetated rocky terrain, and the results were compared. The ExG and GLI indices were generally the most suitable for this purpose, with the best F-scores of 97.7 and 95.4, respectively, and the best-balanced accuracies for the same combination of the method/vegetation index of 98.9 and 98.3%, respectively. Surprisingly, these best results were achieved using the simplest method of threshold determination, considering only a single class (vegetation) with a normal distribution. This algorithm outperformed all other methods, including those based on a support vector machine and a deep neural network. Thanks to its simplicity and ease of use (only several patches representing vegetation must be manually selected as a training set), this method can be recommended for vegetation removal from rocky and anthropogenic surfaces.