Breast cancer is ranked first as the most common cancer case affecting women in the world. Early detection of breast cancer can increase the chances of survival in patients. The role of the radiologist is necessary for the detection of breast cancer, and the radiologists often have limitations in conducting disease consultations with so many patients. The detection gives a subjective result because the process is based on the decision-making of the radiologists. In this work, we proposed a system to detect and classify breast cancer accurately to anticipate delays in patient handling and subjective result. We proposed a digital image processing method using mammograms to classify breast cancer into four categories based on tissue density, namely BI-RADS I, II, III, and IV. The main stages carried out in this research are images processing, feature extraction, data normalization, feature selection, classification, and parameter optimization. This method uses GLCM to extract texture features and two feature selection methods namely, RFE-RF and Chi-Square. The method was tested with various classifiers such as SVM, KNN, Random Forests, and Decision Trees. The hyper-parameters of the classifier were optimized using GridSearch. The final result is measure using accuracy. In this work, Random Forest with the RFE-RF gives the highest accuracy of 99.7%. Feature selection offers a significant impact on improving accuracy. The results of this work prove that our system can classify breast cancer with high accuracy. So that our system can solve problems to assist radiologists in screening mammograms and help make decisions to diagnose patients with breast cancer based on density.