Among various types of cancer, breast cancer is considered as second largest hazardous diseases that cause death. Initially, small lump like structure grow from breast cells which are considered as malignant lumps. To find the available of malignancy in breast area, several checkups such as self-test and periodic has to be done to reduce the death rate due to breast cancer. But, the classification of breast cancer by medical physicians using available techniques is not sufficient, so it is important to improve the classification technique using neural network. The four important phases namely preprocessing, segmentation, feature extraction, and classification can be done in constructed Deep Belief Network (DBN) whereas preprocessing makes to remove the noise and artifacts of mammogram image and then the glands are enhanced. The preprocessed output is given to fuzzy c-means segmentation process with the help of masking. Again the feature extractors such as Scale Invariant Feature Transform (SIFT) and Speeded up Robust Transform (SURF) made to apply on two types of classifiers such as gradient boosting tree classifier and adaboost classifier. The examination is done interms of parameters such as accuracy, precision, recall and F1 score.

Breast cancer continues to be the common cancer which causes more decease among women and about more than two million cases is identified every year and according to the record 523,000 deaths are caused per year due to breast cancer. Mammographic mass identification and segmentation are accomplished usually as sequential and distinct tasks, where in previous studies segmentation was often manually performed only on true positive cases. Machine learning (ML) approaches have been grown from manually provided inputs to systematic initializations. The developments in ML techniques have produced independent and more intelligent computeraided diagnosis (CAD) systems. Moreover, due to the learning ability, ML techniques have been upgraded constantly. Recently, ML techniques are progressive with deeper and varied representation methods, generally termed as deep learning (DL) techniques, and have produced significant impacts on increasing the ability of diagnosing using CAD systems. So this paper proposes the novel architecture in detecting the breast cancer. Here the input database is mammogram image dataset. Initially this image has been resized by pre-processing process, then this pre-processed image has been segmented using Mask RCNN (Region-based convolution neural networks). Then this segmented image has been sent for extracting the feature using inception V3 and ResNet 152 networks. The ensemble classifier of decision tree and random forest has been used for classification and this classified output is detected based on the color variation and tumor size gives the enhanced accuracy in detecting the cancer.