Breast cancer occurs to over 8% women during their lifetime, and is a leading cause of death among women. Sonography is superior to mammography in its ability to detect focal abnormalities in the dense breasts and has no side-effect. In this paper, we proposed a novel automatic segmentation algorithm based on the characteristics of breast tissue and the eliminating particle swarm optimization (EPSO) clustering analysis. The characteristics of mammary gland in breast ultrasound (BUS) images are analyzed and utilized, and a method based on step-down threshold technique is employed to locate the mammary gland area. The EPSO clustering algorithm employes the idea of "survival of the superior and weeding out the inferior". The experimental results demonstrate that the proposed approach can segment BUS image with high accuracy and low computational time. The EPSO clustering method reduces the computational time by 32.75% compared with the standard PSO clustering algorithm. The proposed approach would find wide applications in automatic lesion classification and computer aided diagnosis (CAD) systems of breast cancer.