Accurate mapping of date palm trees is essential for their sustainable management, yield estimation, and environmental studies. In this study, we integrated geographic object-based image analysis, class-specific accuracy measures, fractional factorial design, metaheuristic feature-selection technique, and rule-based classification to detect and map date palm trees from very-high-spatial-resolution (VHSR) aerial images of two study areas. First, multiresolution segmentation was optimized through the synergy of the F1-score accuracy measure and the robust Taguchi design. Second, ant colony optimization (ACO) was adopted to select the most significant features. Out of 31 features, only 12 significant color invariants and textural features were selected. Third, based on the selected features, the rule-based classification with the aid of a decision tree algorithm was applied to extract date palm trees. The proposed methodology was developed on a subset of the first study area, and ultimately applied to the second study area to investigate its efficiency and transferability. To evaluate the proposed classification scheme, various supervised object-based algorithms, namely random forest (RF), support vector machine (SVM), and k-nearest neighbor (k-NN), were applied to the first study area. The result of image segmentation optimization demonstrated that segmentation optimization based on an integrated F1-score class-specific accuracy measure and Taguchi statistical design showed improvement compared with objective function, along with the Taguchi design. Moreover, the result of the feature selection by ACO outperformed, with almost 88% overall accuracy, several feature-selection techniques, such as chi-square, correlation-based feature selection, gain ratio, information gain, support vector machine, and principal component analysis. The integrated framework for palm tree detection outperformed RF, SVM, and k-NN classification algorithms with an overall accuracy of 91.88% and 87.03%, date palm class-specific accuracies of 0.91 and 0.89, and kappa coefficients of 0.90 and 0.85 for the first and second study areas, respectively. The proposed integrated methodology demonstrated a highly efficient and promising tool to detect and map date palm trees from VHSR aerial images.