An unsupervised framework for classifying heart sound data is proposed in this paper. Our goal is to cluster unknown heart sound recordings, such that each cluster contains sound recordings belonging to the same heart diseases or normal heart beat category. This framework is more flexible than the existing supervised classification of heart sounds by the case when heart sound data belong to undefined categories or when there is no prior template data for building a heart sound classifier. To this end, methods are proposed for heart sound feature extraction, similarity computation, cluster generation, and estimation of the optimal number of clusters. Our experiments show that the resulting clusters based on our system are roughly consistent with the heart beat categories defined by human labeling.