How Do Designers Choose Among Multiple Noisy Information Sources in Engineering Design Optimization? An Experimental Study

Abstract

Designers make process-level decisions to (i) select designs for performance evaluation, (ii) select information source, and (iii) decide whether to stop design exploration. These decisions are influenced by problem-related factors, such as costs and uncertainty in information sources, and budget constraints for design evaluations. The objective of this paper is to analyze individuals’ strategies for making process-level decisions under the availability of noisy information sources of different cost and uncertainty, and limited budget. Our approach involves a) conducting a behavioral experiment with an engineering optimization task to collect data on subjects’ decision strategies, b) eliciting their decision strategies using a survey, and c) performing a descriptive analysis to compare elicited strategies and observations from the data. We observe that subjects use specific criteria such as fixed values of attributes, highest prediction of performance, highest uncertainty in performance, and attribute thresholds when making decisions of interest. When subjects have higher budget, they are less likely to evaluate points having highest prediction of performance, and more likely to evaluate points having highest uncertainty in performance. Further, subjects conduct expensive evaluations even when their decisions have not sufficiently converged to the region of maximum performance in the design space and improvements from additional cheap evaluations are large. The implications of the results in identifying deviations from optimal strategies and structuring decisions for further model development are discussed.

Descriptive Models of Sequential Decisions in Engineering Design: An Experimental Study

“…These models are inspired from the literature, observations of past experiments [30], and survey responses in the experiment detailed in Ref. [31].…”

A Reusable and Executable Information Model of Experiments on Human Decision Making in Systems Engineering and Design

“…The decision processes and some vital information of example experiments are shown in Figure 5. More details can be found in Refs [40] and [25] respectively.…”

“…To illustrate the utilization of the DEDS template ontology in terms of capturing knowledge and facilitating experimental design, in this section, we discuss the Decision Strategy Experiment [40], [50] as an example.…”

“…Sample Experiments: (a) Decision Strategy Experiment[40], participants are asked to find the maximum value of an unknown function by given ''tools'' and budget; (b) Domain Knowledge Experiment[25], participants are required to design a roller coaster track with maximum enjoyment.…”