Crop productivity and disaster management can be enhanced by employing hyperspectral images. Hyperspectral imaging is widely utilized in identifying and classifying objects on the ground surface for various agriculture application uses such as crop mapping, flood management, identifying crops damaged due to flood/drought, etc. Hyperspectral imaging-based crop classification is a very challenging task because of spectral dimensions and poor spatial feature representation. Designing efficient feature extraction and dimension reduction techniques can address high dimensionality problems. Nonetheless, achieving good classification accuracies with minimal computation overhead is a challenging task in Hyperspectral imaging-based crop classification. In meeting research challenges, this work presents Hyperspectral imaging-based crop classification using soft-margin decision boundary optimization (SMDBO) based Support Vector Machine (SVM) along with Image Fusion-Recursive Filter (IFRF) and Inherent Feature Extraction (IFE). In this work, IFRF is used for reducing spectral features with meaningful representation. Then, IFE is used for differentiating physical properties and shading elements of different objects spatially. Both spectral and spatial features extracted are trained using SMDBO-SVM for performing hyperspectral object classification. Using SMDBO-SVM for Hyperspectral object classification aid in addressing class imbalance issues; thus, the proposed IFE-SMDBO-SVM model achieves better accuracies and with minimal misclassification in comparison with state-of-art statistical and Deep Learning (DL) based Hyperspectral object classification model using standard dataset Indian Pines and Pavia University.