Interpreting Idea Maps: Pairwise Comparisons Reveal What Makes Ideas Novel

Abstract

Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets to score ideas on their uniqueness. However, their decisions about novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two-dimensional (2D) idea maps. We derive these maps by asking participants for simple similarity comparisons of the form “Is idea A more similar to idea B or to idea C?” We show that these maps give insight into the relationships between ideas and help understand the design domain. We also propose that novel ideas can be identified by finding outliers on these idea maps. To demonstrate our method, we conduct experimental evaluations on two datasets—colored polygons (known answer) and milk frother sketches (unknown answer). We show that idea maps shed light on factors considered by participants in judging idea similarity and the maps are robust to noisy ratings. We also compare physical maps made by participants on a white-board to their computationally generated idea maps to compare how people think about spatial arrangement of design items. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

Examining the Boundary Between Near and Far Design Stimuli

“…For the number of parts in each stimulus set ( 16), a total of 1680 unique triplet trials are possible. Ahmed et al (2019) recommend that a minimum of 30% of the full stimulus set is needed to construct a robust embedding space of human representations. In our study, 36 participants completed 25 triplet ratings for each stimulus set.…”

“…A common task used to elicit these judgments is a triplet rating task where one of two options is selected as being more similar to a given reference. Prior work by Nandy and Goucher-Lambert (2022) andAhmed et al (2019) have used triplet similarity ratings to generate embedding spaces for human representations of design stimuli. Preceding each triplet rating task of 25 trials, participants were told that "In the [first/second] section of this study, you will consider the [function/appearance] of parts when assessing similarity".…”

Adaptive Inspirational Design Stimuli: Using Design Output to Computationally Search for Stimuli That Impact Concept Generation

“…Ahmed et al provide a similar, but more precise definition for the utility of a design as: "a measure of the designs' performance and can depend on multiple domain dependent factors like functionality, feasibility, usefulness, impact, investment potential, scalability, etc." [30]. Some works have used less specific derivatives.…”