Portugal has a relevant role in natural stone production and trading. The national sector has been trying to follow the requirements of the Fourth Industrial Era, where automation of stone processing operations is necessary to increase productivity, balance the decreasing workforce and reduce stone waste. The identification of defects or defective areas in stone slabs is a time-consuming operation: not only highly subjective, dependent on the operator experience, but also disconnected from geological knowledge. Despite recent advances in automatic pattern recognition of stone products’ surface, stone selection lacks a geological foundation. Thus, this paper aims to discuss the terminology of stone defects and singularities applied to stone materials and in the stone selection process. A classification system of natural stone heterogeneities and discontinuities is developed to be used by stone operators and to be applied in future image analysis research. The system, based on three main classifiers (colour heterogeneities, textural and structural heterogeneities, and discontinuities) is validated by seventeen different Portuguese stone varieties and appears to be a suitable qualitative descriptor of stone slabs and tiles, showing flexibility in its use.