Oral cancer, also known as oral squamous cell carcinoma (OSCC), is one of the most prevalent types of cancer and caused 177,757 deaths worldwide in 2020, as reported by the World Health Organization. Early detection and identification of OSCC are highly correlated with survival rates. Therefore, this study presents an automatic image-processing-based machine learning approach for OSCC detection. Histopathological images were used to compute deep features using various pretrained models. Based on the classification performance, the best features (ResNet-101 and EfficientNet-b0) were merged using the canonical correlation feature fusion approach, resulting in an enhanced classification performance. Additionally, the binary-improved Haris Hawks optimization (b-IHHO) algorithm was used to eliminate redundant features and further enhance the classification performance, leading to a high classification rate of 97.78% for OSCC. The b-IHHO trained the k-nearest neighbors model with an average feature vector size of only 899. A comparison with other wrapper-based feature selection approaches showed that the b-IHHO results were statistically more stable, reliable, and significant (p < 0.01). Moreover, comparisons with those other state-of-the-art (SOTA) approaches indicated that the b-IHHO model offered better results, suggesting that the proposed framework may be applicable in clinical settings to aid doctors in OSCC detection.