Segmentation is a technique for separating an image into discrete areas in order to separate objects of interest from their surroundings. In image analysis, segmentation—which encompasses detection, feature extraction, classification, and treatment—is crucial. In order to plan treatments, segmentation aids doctors in measuring the amount of tissue in the breast. Categorizing the input data into two groups that are mutually exclusive is the aim of a binary classification problem. In this case, the training data is labeled in a binary format based on the problem being solved. Identifying breast lumps accurately in mammography pictures is essential for the purpose of prenatal testing for breast cancer. The proposed TLA (Transfer Learning Approach) based CNN (Convolution Neural Network) –TLA based CNN aims to offer binary classification for rapid and precise breast cancer diagnosis (benign and malignant). In order to predict the sub-type of cancer, this exploration as used Deep Learning techniques on the Histogram of Oriented Gradient (HOG) - Feature extraction technique that creates a local histogram of the image to extract features from each place in the image with CNN classifier. This research work employs two well-known pre-trained models, ResNet-50 and VGG16, to extract characteristics from mammography images. The high-level features from the Mammogram dataset are extracted using a transfer learning model based on Visual Geometry Group (VGG) with 16-layer and Residual Neural Network with 50-layers deep model architecture (ResNet-50). The proposed model TLA based CNN has achieved 96.49% and 95.48% accuracy as compared to ResNet50 and VGG16 in the breast cancer classification and segmentation.