Evaluation of Idea Generation Methods for Conceptual Design: Effectiveness Metrics and Design of Experiments

Abstract

A wide range of formal methods have been devised and used for idea generation in conceptual design. Experimental evidence is needed to support claims regarding the effectiveness of these methods in promoting idea generation in engineering design. Towards that goal this paper presents a set of effectiveness metrics experimental methods, data collection and analysis techniques. Statistically based Design of Experiments (DOE) principles were used in developing the guidelines. Four classes of operating variables were considered to characterize the design problem and the environment. The effectiveness metrics proposed are based on outcome and consists of the quantity, quality, novelty, and variety of ideas generated. Two experimental approaches have been developed. In the Direct Method, the influence of the type of design problem and various parameters related to the procedure of an idea generation method is measured by using the method in its entirety. In the Indirect Method, each idea generation method is decomposed into key components and its overall effectiveness is predicted by experimentally studying the effectiveness of its components and their mutual interactions. [S1050-0472(00)02004-3]

Evaluation of a method and a computer tool for generating concept designs

“…Earlier, a similar observation was also expressed by Shah et al (2000Shah et al ( , 2003, who found that very little formal experimental evidence exists to indicate usefulness of idea generation methods for conceptual design and design methods/tools in general (Ruder and Sobek II 2007). The knowledge and ability to use specific methods and tools (especially for systematic synthesis in design) constitute the competency of engineering graduates (Eder and Hubka 2005).…”

“…The approach to conduct the experiment basically follows the 'Direct Method' (more specifically, the 'Intrinsic Merit Stage') as proposed by Shah et al (2000): formation of two groups (i.e. control and experimental group), selection of a design task, generation of results and the type of data collected.…”

“…A concept design is a solution that has the potential to provide the main function as described in the design task. It is an important comparative item (Shah et al 2000), because there is a positive correlation between the number of generated alternative product concepts and their quality (Parnes 1961, Cross and Cross 1996, Andreasen and Hein 2000.…”

“…According to Shah et al (2000Shah et al ( , 2003, there are two fundamental measures to evaluate the usefulness of a conceptual design method: (i) effectiveness of expanding the design space and (ii) thoroughness of exploring the design space. They proposed quantity, quality, novelty and variety as specific measures; to begin with, quantity and variety were adopted in our experiment.…”

Changing Creativity through Engineering Education and Bio-Inspired Design

“…These metrics were introduced by Shah et al 31 and further developed by Linsey et al 32,33 . These metrics are used to quantitatively compare the experimental conditions based on the solutions generated by the participants.…”