Down jacket fabric is greatly important in determining the quality of a down jacket. In order to enrich the research on fabric handle, subjective and objective evaluations were made for down jacket fabrics that were less studied. The comprehensive handle evaluation system for fabrics and yarns (CHES-FY) can be used to evaluate the tactile handle of the fabric by accurately and efficiently measuring the basic mechanical properties of the fabric. Therefore, the CHES-FY was used to link the objective evaluation with the subjective handle, so as to effectively estimate the total handle value of the down jacket fabric. Fifty-two kinds of down jacket fabrics were objectively tested through measuring 17 extracted parameters, and principal component analysis was adopted to establish the five main handle characteristics of fullness, softness, stiffness, smoothness, looseness and tightness to characterize basic style of the down jacket fabrics. The results showed that the subjective and objective results were in good agreement. These characteristics can be used as indicators to characterize fabric performance, and the principal component expression to characterize fabric handle can better predict the handle characteristics of down jacket fabrics. This also proves that the CHES-FY can quickly and accurately obtain the fabric handle value, and can also evaluate the fabric quality level.