Five multiattribute weighting methods were compared in an Internet experiment. This is the first experiment where the subjects created the alternatives and attributes themselves. Each subject used five methods to assess attribute weights, one version of the Analytic Hierarchy Process (AHP), DIRECT weighting, Simple Multiattribute Rating Technique (SMART), SWING weighting and TRADEOFF weighting. They can all be used following the principles of multiattribute value theory. Furthermore, each of them asks decision makers to give numerical estimates of weight ratios although the elicitation questions are different. In earlier studies, however, these methods have yielded different weights. Our results suggest that weights are different because the methods explicitly or implicitly lead the decision makers to choose their responses from a limited set of numbers. The other consequences are that the spread of weights and the inconsistency between the preference statements depend on the number of attributes that a decision maker considers simultaneously.