In this proposed work, we presented a system to classify the object. Firstly, the given images are segmented using Region merging Segmentation method. Later the background eliminated images are divided into number of blocks viz., 4, 16, 32. The features like Scale Invariant Feature Transform (SIFT) and Histogram of Gradients (HOG) are extracted from divided blocks of size 4, 16, 32. To measure the strength of proposed method we compare the Classification vs Retrieval using Support Vector Machine and KD Tree. We conducted the experimentation on Caltech 101 data set. To study the effect of accuracy in classification we pick images from database randomly. The Performance revels that the SVM achieves good performance.