Content-based image retrieval (CBIR) is an research area over the past years that has attracted research. In various medical applications like mammogram analysis CBIR techniques helps the medical team to get similar set of images from a large medical records to help in diagnosis of a disease. This paper proposes an efficient Content-Based Mammogram Image Retrieval method by using an Optimized Classifier. Initially, the input dataset is preprocessed, in which noise removal and contrast enhancement are done. Next, pectoral muscles of the mammogram images are removed using Single Sided Edge Marking (SSEM). Now, feature extraction is done, in which GLCM features, Gabor features and the Local Pattern with Binary features are being removed. The features that are being removed are classified into three classes namely benign, malignant and normal. An optimized classifier named as Adaptive Neuro Fuzzy Inference System (ANFIS), which is optimized by using the Improved Particle Swarm Optimization (IPSO) technique, is used for classification purpose. Finally, similarity is assessed between the trained feature distance vectors and the feature distance vectors of the input query image.Similarity assessment is done using Euclidean Distance metric and the image that has the lowest distance compared with the query is retrieved. The experimental results are obtained for the proposed system and they are compared with the existing techniques.