Among women, breast cancer is one of the most commonly occurring cancers besides skin and cervix cancer. Developing countries are at a higher risk of mortality due to late-stage presentation, inaccessible diagnosis, and treatment. Thermography-based technology, aided with machine learning, for screening/diagnosing breast cancer is non-invasive, cost-wise appropriate, and requires very little equipment in rural areas with limited facilities. In this paper, we systematically compare the state-of-the-art feature extraction approaches on a uniform platform, using two Common datasets, three Feature Selection methods, four well-known Classifiers, and three Cross-Validation strategies and analyze the results, for a fair comparison. Also, we evaluated the performance when all the features were combined (Unified Model) on the same platform. Experimental results show that the classification accuracy improves considerably with the use of feature selection methods. Among all the combinations considered, the classification model with Union_FeatureSet and mRMR gave the best performance for both datasets. We obtained a feature subset of 26 and 34 features (from Union_FeatureSet) with a combination of mRMR and SVM, which are relevant, non-redundant, and distinguish normal and abnormal thermal patterns with the accuracy of 95.73% on the DMR-IR dataset and 92.533% on the RGC-IR dataset.