<p>All over the world breast cancer is a major disease which mostly affects the women and it may also cause death if it is not diagnosed in its early stage. But nowadays, several screening methods like magnetic resonance imaging (MRI), ultrasound imaging, thermography and mammography are available to detect the breast cancer. In this article mammography images are used to detect the breast cancer. In mammography image the cancerous lumps/microcalcifications are seen to be tiny with low contrast therefore it is difficult for the doctors/radiologist to detect it. Hence, to help the doctors/radiologist a novel system based on deep neural network is introduced in this article that detects the cancerous lumps/microcalcifications automatically from the mammogram images. The system acquires the mammographic images from the mammographic image analysis society (MIAS) data set. After pre-processing these images by 2D median image filter, cancerous features are extracted from the images by the hybridization of convolutional neural network with rat swarm optimization algorithm. Finally, the breast cancer patients are classified by integrating random forest with arithmetic optimization algorithm. This system identifies the breast cancer patients accurately and its performance is relatively high compared to other approaches.</p>