Evaluating Crowdsourced Design Concepts With Machine Learning

Abstract

Automation has enabled design of increasingly complex products, services, and systems. Advanced technology enables designers to automate repetitive tasks in earlier design phases, even high level conceptual ideation. One particularly repetitive task in ideation is to process the large concept sets that can be developed through crowdsourcing. This paper introduces a method for filtering, categorizing, and rating large sets of design concepts. It leverages unsupervised machine learning (ML) trained on open source databases. Input design concepts are written in natural language. The concepts are not pre-tagged, structured or processed in any way which requires human intervention. Nor does the approach require dedicated training on a sample set of designs. Concepts are assessed at the sentence level via a mixture of named entity tagging (keywords) through contextual sense recognition and topic tagging (sentence topic) through probabilistic mapping to a knowledge graph. The method also includes a filtering strategy, the introduction of two metrics, and a selection strategy for assessing design concepts. The metrics are analogous to the design creativity metrics novelty, level of detail, and a selection strategy. To test the method, four ideation cases were studied; over 4,000 concepts were generated and evaluated. Analyses include: asymptotic convergence analysis; a predictive industry case study; and a dominance test between several approaches to selection of high ranking concepts. Notably, in a series of binary comparisons between concepts that were selected from the entire set by a time limited human versus those with the highest ML metric scores, the ML selected concepts were dominant.

A Data-Driven Approach for Creative Concept Generation and Evaluation

“…The indication is based on prior human experiments (Luo et al, 2018) as well as big data experiments (Alstott et al, 2017). With regards to using idea distances for concept evaluation, other semantic measurement tools, such as SEMILAR and TechNet, as well as several existing computational idea evaluation methods such as the InnoGPS and the machine learning-based concept evaluation method proposed by Camburn et al (2019;2020), which have employed the use of semantic distances, could be used. However, further research is needed to explore the 'definition' of far-related and closely-related ideas in computational measurements.…”

Dynamics of automatized measures of creativity: mapping the landscape to quantify creative ideation

“…Concerning the three categories of text similarity, namely, string similarity, corpus-based similarity, and knowledge-based similarity as set out in Table 3, the scoping review shows differences in the process of similarity computation that have an impact on how they are applied. On the one hand, string-based and knowledge-based similarities have limited application in automatic creativity evaluation because string-based only considers syntactic similarity (not semantic) and knowledge-based only extracts from text-specific entities, such as a person's name, place, and money (Camburn et al, 2019). During ideation, the knowledge-based might focus on entities rather than technical terms or scientific jargon within the sentence used by sentences solving a scientific challenge.…”

“…Despite this high interest, our revision indicates multifariousness in defining and measuring novelty. As a consequence of that, the reviewed studies refer to novelty using the following different words or manifestations, namely, (1) uniqueness: the uniqueness of a concept related to the other concepts (Camburn et al, 2019);…”

“…Considering the above implications of NLP, current research focuses on studying how different computational techniques can measure creativity dimensions (Doboli et al, 2020). Research on this topic has been very productive and has designed other computational techniques to measure creativity dimensions, e.g., (1) novelty is measured by keyword similarity (Prasch et al, 2020), part of speech tagging (Karampiperis et al, 2014;Camburn et al, 2019), and different ML classifiers, such as Bayesian classifiers, random tree, and Support Vector Machine (SVM) (Manske and Hoppe, 2014;Simpson et al, 2019;Doboli et al, 2020); (2) originality dimension is measured by Latent Semantic Analysis (LSA) (Dunbar and Forster, 2009), Global Vectors for word representation (GloVe) (Dumas et al, 2021), and part of speech tagging (Georgiev and Casakin, 2019); (3) fluency dimension is measured by LSA (Dumas and Dunbar, 2014;LaVoie et al, 2020); (4) elaboration dimension is measured by parts of speech tagging (Dumas et al, 2021); and (5) level of details dimension is measured by text-mining methods (Camburn et al, 2019).…”

Semantic Networks for Engineering Design: A Survey

“…Chen and Krishnamurthy (2020) proposed an interactive procedure to retrieve words and terms in ConceptNet to inspire designers. Camburn et al (2019) proposed a set of new metrics for automatic evaluation of the natural language descriptions of a large number of crowdsourced design ideas, and their evaluation was based on the Freebase (Bollacker et al, 2008), another large knowledge database managed by Google. These engineering design studies generally rely on common-sense knowledge bases, such as WordNet and ConceptNet, or language models not trained specifically for engineering.…”