Distance measure is an essential tool to characterize the difference between two samples. Recently, lots of distance measures have been proposed for hesitant fuzzy sets (HFSs). In this paper, we shall propose some novel distance formulas to measure the deviation between two HFSs. First, we define some new concepts including the hesitant fuzzy variance, covariance, and correlation coefficient. Based on these concepts and the idea of the traditional Mahalanobis distance, the hesitant Mahalanobis distance between two HFSs is developed. Then we discuss the properties of the new distance measure and uncover the significant characteristic of the introduced distance measure that it can give the attributes an adaptive weight and can eliminate the influence of the correlation between the attributes under hesitant fuzzy environment. And then, some extensions of this new distance measure are also developed. Second, to show the validity and applicability of the proposed distance measures, we compare them with the existing ones in decision making and cluster analysis with some numerical examples. Third, using the proposed distance measures, we develop two algorithms to estimate the optimal number of clusters, which is a new application area of the hesitant fuzzy distance measures. Finally, the two algorithms are applied in a numerical example to illustrate their applicability and efficiency.