A wide range of formal methods have been devised and used for idea generation in conceptual design. Experimental evidence is needed to support claims regarding the effectiveness of these methods in promoting idea generation in engineering design. Towards that goal this paper presents a set of effectiveness metrics experimental methods, data collection and analysis techniques. Statistically based Design of Experiments (DOE) principles were used in developing the guidelines. Four classes of operating variables were considered to characterize the design problem and the environment. The effectiveness metrics proposed are based on outcome and consists of the quantity, quality, novelty, and variety of ideas generated. Two experimental approaches have been developed. In the Direct Method, the influence of the type of design problem and various parameters related to the procedure of an idea generation method is measured by using the method in its entirety. In the Indirect Method, each idea generation method is decomposed into key components and its overall effectiveness is predicted by experimentally studying the effectiveness of its components and their mutual interactions. [S1050-0472(00)02004-3]

This paper presents the development and evaluation of a technique (C‐Sketch) for concept generation in a collaborative engineering design setting. The paper reviews both the intrinsic merit of C‐Sketch, as well as relative merit compared to other techniques in the same class. This analysis is based on results from experiments conducted over five years. Both the process and outcome were evaluated, with greater emphasis on the latter. This study found that C‐Sketch not only has intrinsic merit, but also measures higher in all outcomes when compared to Method 6‐3‐5. Also, C‐Sketch was at least as good as the Gallery Method in the quality of ideas produced and better in variety and novelty of ideas. This paper is a consolidation of all empirical studies related to C‐Sketch.

Several formal Idea Generation (IG) methods can be found in the design literature ranging from simple methods like brainstorming to complex procedures such as synectics. Experimental evidence is needed to support claims regarding the effectiveness of these methods in promoting idea generation. This paper attempts to prescribe a method for evaluating intuitive idea generation methods. The set of guidelines for experimental evaluation include metrics, how to measure against those metrics (quality, quantity, novelty, variety), how to compute the results, and types of experiments to run. Statistically based Design of Experiments (DOE) principles are in developing the guidelines. Four classes of variables are considered. Two experiment methodologies are proposed. In the Direct Method experiments are conducted using the idea generation method in its entirety. In the Indirect Method, each idea generation method is decomposed into key components and its effectiveness is predicted by experimentally studying the effectiveness of its components and interactions between them.