Mammography is a non-invasive method to study breast tissues for abnormalities. Computer-aided diagnosis (CAD) can automate the process of diagnosing malignant and benign tumors accurately. However, accurate results can be hampered by the presence of the pectoral muscle, which has a similar opacity to the breast tissue area. Detecting and removing pectoral muscles is not trivial due to various factors, and there are artifacts present near the pectoral muscle that can hamper proper segmentation. Given the significance of the topic, it is crucial to devise an accurate method for automatically detecting the muscle area in a mammography image and eliminating it from the rest of the image. This process of removing the pectoral muscle from the breast image can aid in precise segmentation and diagnosis of the tumor area, ultimately leading to faster diagnosis and better outcomes for patients. This study examined two segmentation algorithms, Level Set and Region Growing, for segmenting the pectoral muscle. An Improved Region Growingbased (IRG) algorithm was also proposed and showed promising results in automatically segmenting the pectoral muscle. All algorithms were tested on the MIAS dataset, and radiologists evaluated the results, showing an accuracy rating of up to 83% for IRG. The results indicated that IRG outperformed Level Set considerably due to many optimizations and modifications. IRG can be used as part of the preprocessing unit of an automated cancer diagnosis system.