Brain tumor is a fatal disease, caused by the growth of abnormal cells in the brain tissues. Therefore, early and accurate detection of this disease can save patient’s life. This paper proposes a novel framework for the detection of brain tumor using magnetic resonance (MR) images. The framework is based on the fully convolutional neural network (FCNN) and transfer learning techniques. The proposed framework has five stages which are preprocessing, skull stripping, CNN-based tumor segmentation, postprocessing, and transfer learning-based brain tumor binary classification. In preprocessing, the MR images are filtered to eliminate the noise and are improve the contrast. For segmentation of brain tumor images, the proposed CNN architecture is used, and for postprocessing, the global threshold technique is utilized to eliminate small nontumor regions that enhanced segmentation results. In classification, GoogleNet model is employed on three publicly available datasets. The experimental results depict that the proposed method is achieved average accuracies of 96.50%, 97.50%, and 98% for segmentation and 96.49%, 97.31%, and 98.79% for classification of brain tumor on BRATS2018, BRATS2019, and BRATS2020 datasets, respectively. The outcomes demonstrate that the proposed framework is effective and efficient that attained high performance on BRATS2020 dataset than the other two datasets. According to the experimentation results, the proposed framework outperforms other recent studies in the literature. In addition, this research will uphold doctors and clinicians for automatic diagnosis of brain tumor disease.