Accurate and timely diagnosis is critical for effective medical intervention in brain-related diseases, including Brain Tumors and Alzheimer's. Most current state-of-the-art (SOTA) approaches in medical imaging focus on diagnosing a single brain disease at a time. However, recent research has unveiled the intricate connections between various brain diseases, with the realization that treating one condition may lead to the development of others. Consequently, there is a growing need for accurate diagnostic systems that address multiple brain-related diseases. However, training separate models for different diseases can impose substantial computational overhead. To address this challenge, our paper introduces BrainNet, an innovative neural network architecture explicitly tailored for classifying brain images. Our primary objective is to propose a single, robust framework capable of diagnosing a spectrum of brain-related diseases. We present a comprehensive validation of BrainNet's efficacy, specifically in diagnosing Brain Tumors and Alzheimer's disease. Remarkably, our proposed model workflow surpasses the current state-of-the-art (SOTA) methods, demonstrating a substantial enhancement in accuracy and precision. Furthermore, it maintains a balanced performance across different classes in the Brain Tumor and Alzheimer's dataset, emphasizing the versatility of our architecture for precise disease diagnosis. Thus, this paper marks a significant stride toward a unified solution for diagnosing diverse brain-related diseases