Interpreting Measures of Rarity and Novelty: Investigating Correlations Between Relative Infrequency and Perceived Ratings

Abstract

Central to the struggle in design ideation research is the quantification of abstract and qualitative measures. Among these measures, creativity, originality, and novelty are some of the most subjective and generally disagreed upon constructs. In recent years in the design community, novelty has primarily been measured with two distinct styles of metrics: relative infrequency and perceived ratings. Relative infrequency captures how rare an idea is within an idea set for an objective representation of novelty, while ratings quantify the perceptions of appropriate judges for an intuitive understanding of novelty. This paper investigates the convergent validity between these two styles through the implementation of three previously published methods. Moderate convergent validity is shown between a measure of relative infrequency and perceived ratings leading to clear and meaningful recommendations for metrics in future research. However, the degree of disagreement warrants differentiation between the two styles regarding terminology and analysis. This study will aid in the research and interpretation of future studies of creativity in ideation.

Refined metric for a-posteriori novelty assessments

“…In particular, it is possible to find studies focused on possible correlations between subjective metrics and uncommonness metrics (e.g. Hocevar [1979] and the more recent comparison performed by Weaver, Caldwell and Sheafer [2019]). Also Shah, Vargas-Hernandez, and Smith (2003) performed a similar comparison in order to evaluate their metrics.…”

“…More in particular, this work can be positioned in the set of contributions focused on design creativity metrics (e.g. Hay et al 2020;Jagtap, 2019;Kershaw et al 2019;Lopez-Mesa and Vidal 2006;Nelson et al 2009;Oman et al 2013;Sarkar and Chakrabarti 2011;Shah, Vargas-Hernandez, and Smith 2003;Verhaegen et al 2013;Weaver, Caldwell, and Sheafer 2019).…”