Brain-computer interface (BCI) has extensively been used for rehabilitation purposes.Being in the research phase, the brainwave based wheelchair controlled systems suffer from several limitations, e.g., lack of focus on mental activity, complexity in neural behavior in different conditions, and lower accuracy. Being sensitive to the color stimuli, the EEG signal changes promises a better detection. Utilizing the Electroencephalogram (EEG changes due to different color stimuli, a methodology of wheelchair controlled by brainwaves has been presented in this study. Red, Green, Blue (primary colors) and Yellow (secondary color) were chosen as the color stimuli and utilized in a 2 × 2 color window for four-direction command, namely left and right, forward and stop. Alpha, beta, delta and theta EEG rhythms were analyzed, time and frequency domain features were extracted to find the most influential rhythm and accurate classification model. Four classifiers, namely, K-Nearest Neighbor (KNN), Support Vector Machine (SVM), Random Forest Classifier (RFC) and Artificial Neural Networks (ANN) were trained and tested for assessing the performance of each of the EEG rhythm, with a five-fold cross-validation. Four different performance measures, i.e. sensitivity, specificity, accuracy and area under the receiver operating characteristic curve were utilized to examine the wholescale performance. The results suggested that Beta EEG rhythm performs the best apart from all the rhythms for the color stimuli based wheelchair control. While comparing the performance of the classifiers, ANN-based classifier shows the best accuracy of 82.5%, which is higher than the performance of the three other classifiers.