Yellow jadeite is a kind of jadeite caused by secondary leaches. With a certain degree of transparency, its color is usually influenced by the objective conditions including light source and background. In this paper, 19 Munsell neutral value scales were selected to study the impact caused by background under the standard light source D65, where we found that the N9.5 Value Scale is suitable for color grading. All 33 yellow jadeites were tested by UV-VIS spectrophotometer and colorimetric analysis, and the colors were graded by the affinity propagation (AP) and K-means clustering methods. According to the result, the yellow jadeite color was classified into Fancy, Fancy Intense, Fancy Deep, and Fancy Light. It is also concluded that AP is more feasible to provide the color grading scheme, while K-means can be applied to the optimization of choosing clustering center.