A brain tumor is the uncharacteristic progression of tissues in the brain. These are very deadly, and if it is not diagnosed at an early stage, it might shorten the affected patient's life span. Hence, their classification and detection play a critical role in treatment. Traditional Brain tumor detection is done by biopsy which is quite challenging. It is usually not preferred at an early stage of the disease. The detection involves Magnetic Resonance Imaging (MRI), which is essential for evaluating the tumor. This paper aims to identify and detect brain tumors based on their location in the brain. In order to achieve this, the paper proposes a model that uses an extended deep Convolutional Neural Network (CNN) named Contour Extraction based Extended EfficientNet-B0 (CE-EEN-B0) which is a feed-forward neural network with the efficient net layers; three convolutional layers and max-pooling layers; and finally, the global average pooling layer. The site of tumors in the brain is one feature that determines its effect on the functioning of an individual. Thus, this CNN architecture classifies brain tumors into four categories: No tumor, Pituitary tumor, Meningioma tumor, and Glioma tumor. This network provides an accuracy of 97.24%, a precision of 96.65%, and an F1 score of 96.86% which is better than already existing pre-trained networks and aims to help health professionals to cross-diagnose an MRI image. This model will undoubtedly reduce the complications in detection and aid radiologists without taking invasive steps.