Breast cancer is more common in women and the mortality rate also increases in recent days. Early detection of breast cancer is reduces the severity of the disease to some extent. Various image processing and classification techniques imposed on a particular image to detect and diagnose the breast cancer clearly. For early detection, an efficient methodology is needed. To reduce death rate, accurate discovery of the disease should be efficiently implemented. The mask R-CNN is used for segmentation to identify abnormalities and ensemble CNN is used to classify the benign and malignant tumour from the given mammographic breast cancer image. The input is fed into the enhanced fuzzy based median filter which is used to remove speckle noises which in turn increases the clarity of images. The noise-free images are fed into the proposed mask R-CNN for segmentation. It is an effective framework for segmentation of medical images. The classification process is then given into ensemble based CNN classifier for prediction of mammogram images. The ensemble based CNN is proposed for the classification of mammograms as benign and malignant. The proposed method focus on the optimization which is performed to minimize the memory requirements and running time while maintaining the high performance of the classifier. The performance metrics such as accuracy, precision, f-measure, and recall is evaluated to measure the efficacy of the proposed design. Using MIAS and DDSM dataset the performance metrics has been evaluated and compared with existing approaches. The results shows that the performance of the proposed method shows higher efficiency.