The mammogram images are exposed to region of interest algorithm for extraction of infected area. The region of interest algorithm is fine-tuned using the combination of red, green and blue color matrices thresholds. The threshold limits are discussed in later section of the paper. The features including spot size, shape, brightness, contrast, density and Zernike moments are used to generate a vector set that is used to retrieve the images from the data base. The data base used for CBIR system for mammograms is MIAS data base consisting of different class of mammogram images. The data base contains images in PGM format and is converted to jpeg format using the matlab function class. The database images are categorized into different class as per the database instruction. Equal no. of images from each class are used to train a support vector machine classifier. Rests of the images are used to test the algorithm.