Due to rapid growth of on-line information, text classification has become one of key technique for handling and organizing text data. One of the reasons to build taxonomy of documents is to make it easier to find relevant documents, content filtering and topic tracking.LS-SVM is the classifier, used in this paper for efficient classification of text documents. Text data is normally highdimensional characteristic, to reduce the high-dimensionality also possible with SVM.In this paper we are improving classification accuracy and dimensionality reduction of a large text data by Least Square Support Vector Machines along with Singular Value Decomposition.