Breast Cancer is one of the fastest growing cancer that causes women to death in the world. The early detection of breast cancer improves the chances of its cure. The malignant tumor that is the sign of breast cancer can be detected by mammography. This paper develops a technique to classify the mammogram images as normal, benign or malignant. This paper applies HAPGD (Hybrid ACO (Ant Colony Optimization), PSO (Particle Swarm Optimization), GA (Genetic Algorithm), and DE (Differential Evolution)) classification algorithm to texture features extracted from the mammogram image. The analysis has been done on the DDSM and MIAS dataset by using classification accuracy, specificity, and sensitivity as the parameter with three state of art algorithms i.e. SVM classifier (without any optimization technique), Firefly (SVM with Firefly optimization), ACO-PSO-GA (SVM with hybrid ACO-PSO-GA optimization). The improvement in the performance measures against three state of art techniques shows the significance of the algorithm.