Since the brain lesion detection and classification is a vital diagnosis task, in this paper, the problem of brain magnetic resonance imaging (MRI) classification is investigated. Recent advantages in machine learning and deep learning allows the researchers to develop the robust computer-aided diagnosis (CAD) tools for classification of brain lesions. Feature extraction is an essential step in any machine learning scheme. Time-frequency analysis methods provide localized information that makes them more attractive for image classification applications. Owing to the advantages of two-dimensional discrete orthonormal Stockwell transform (2D DOST), we propose to use it to extract the efficient features from brain MRIs and obtain the feature matrix. Since there are some irrelevant features, twodirectional two-dimensional principal component analysis ((2D) 2 PCA) is used to reduce the dimension of the feature matrix. Finally, convolution neural networks (CNNs) are designed and trained for MRI classification. Simulation results indicate that the proposed CAD tool outperforms the recently introduced ones and can efficiently diagnose the MRI scans.