The analytic hierarchy process has been widely used to determine subjective weights of materials selection criteria in materials selection using multi-criteria decision-making. However, the analytic hierarchy process has some drawbacks: it is difficult to construct a pairwise comparison matrix and meet the consistency requirement. First, we propose a new simplest questionnaire to perform the pairwise comparison without confusion, conventionally and easily. Next, we propose an improved modifying method for inconsistent pairwise comparison matrix according to the following principles: (1) the elements of the reconstructed pairwise comparison matrix should be nine-point scales, (2) the number and modifying the amount of the modified elements should be as small as possible and (3) the deviation between the original and reconstructed pairwise comparison matrixes should be as small as possible. The outline of the proposed modifying method is as follows: (1) calculate the consistency ration decrements of all the pairwise comparison matrixes reconstructed by modifying every element of the original pairwise comparison matrix to the lower and upper adjacent nine-point scales and (2) find the element with the maximum consistency ratio decrement and modify it to the lower or upper adjacent scale. To illustrate the effectiveness, we apply the proposed methods to determine the criteria weights for selecting the best phase change material used in a solar domestic hot water system, and apply the proposed modifying method to some examples from the published papers, and compare the performances with some previous methods. The simplest questionnaire and improved modifying method help materials designers and engineers to apply the analytic hierarchy process method in materials design and optimization problems, much more actively.