The color histogram is a statistical behavior for robust pattern search or matching; however, difficulties have arisen in using it to discriminate among similar objects. Our method, called absent color indexing (ABC), describes how to use absent or minor colors as a feature in order to solve problems while robustly recognizing images, even those with similar color features. The proposed approach separates a source color histogram into apparent (AP) and absent (AB) color histograms in order to provide a fair way of focusing on the major and minor contributions together. A threshold for this separation is automatically obtained from the mean color histogram by considering the statistical significance of the absent colors. After these have been separated, an inversion operation is performed to reinforce the weight of AB. In order to balance the contributions of the two histograms, four similarity measures are utilized as candidates for combination with ABC. We tested the performance of ABC in terms of the F-measure using different similarity measures, and the results show that it is able to achieve values greater than 0.95. Experiments on Mondrian random patterns verify the ability of ABC to distinguish similar objects by margin. The results of extensive experiments on real-world images and open databases are presented here in order to demonstrate that the performance of our relatively simple algorithm remained robust even in difficult cases.