Solid wood panels are widely used in the wood flooring and furniture industries, and paneling is an excellent material for indoor decoration. The classification of colors helps to improve the appearance of wood products assembled from multiple panels due to the differences in surface colors of solid wood panels. Traditional wood surface color classification mainly depends on workers’ visual observations, and manual color classification is prone to visual fatigue and quality instability. In order to reduce labor costs of sorting and to improve production efficiency, in this study, we introduced machine vision technology and an unsupervised learning technique. First-order color moments, second-order color moments, and color histogram peaks were selected to extract feature vectors and to realize data dimension reduction. The feature vector set was divided into different clusters by the K-means algorithm to achieve color classification and, thus, the solid wood panels with similar surface color were classified into one category. Furthermore, during twice clustering based on second-order color moment, texture recognition was realized on the basis of color classification. A sample of beech wood was selected as the research object, not only was color classification completed, but texture recognition was also realized. The experimental results verified the effectiveness of the technical proposal.