The low-level feature, such as Local Directional Pattern (LDP) was used to describe textures and shapes in the image. The advantage of the LDP feature is its robustness under random noise and illumination/light changes. This paper proposed a new approach to classifying and recognizing types of clothing by using Speeded-Up Robust Features (SURF) and Local Directional Pattern (LDP) based on Bag of Features (BoF) model. The key processes of the proposed system are firstly, the human are located and segmented clothing in the image. Secondly, Speeded-Up Robust Features (SURF) is used for detecting the interesting points and LDP features are used to create a codebook. Finally, a support vector machine (SVM) is used to classify the types of clothing. The dataset consists of seven categories of clothing such as sweaters, suits and shirts. Our dataset consists of total 1131 images out of which the training set is 991 images and the remainder is the testing set. The result of the recognition rate achieves an average F-score of 63.36%.