Understanding the Effect of Task Complexity and Problem-Solving Skills on the Design Performance of Agents in Systems Engineering

Abstract

Systems engineering processes coordinate the efforts of many individuals to design a complex system. However, the goals of the involved individuals do not necessarily align with the system-level goals. Everyone, including managers, systems engineers, subsystem engineers, component designers, and contractors, is self-interested. It is not currently understood how this discrepancy between organizational and personal goals affects the outcome of complex systems engineering processes. To answer this question, we need a systems engineering theory that accounts for human behavior. Such a theory can be ideally expressed as a dynamic hierarchical network game of incomplete information. The nodes of this network represent individual agents and the edges the transfer of information and incentives. All agents decide independently on how much effort they should devote to a delegated task by maximizing their expected utility; the expectation is over their beliefs about the actions of all other individuals and the moves of nature. An essential component of such a model is the quality function, defined as the map between an agent’s effort and the quality of their job outcome. In the economics literature, the quality function is assumed to be a linear function of effort with additive Gaussian noise. This simplistic assumption ignores two critical factors relevant to systems engineering: (1) the complexity of the design task, and (2) the problem-solving skills of the agent. Systems engineers establish their beliefs about these two factors through years of job experience. In this paper, we encode these beliefs in clear mathematical statements about the form of the quality function. Our approach proceeds in two steps: (1) we construct a generative stochastic model of the delegated task, and (2) we develop a reduced order representation suitable for use in a more extensive game-theoretic model of a systems engineering process. Focusing on the early design stages of a systems engineering process, we model the design task as a function maximization problem and, thus, we associate the systems engineer’s beliefs about the complexity of the task with their beliefs about the complexity of the function being maximized. Furthermore, we associate an agent’s problem solving-skills with the strategy they use to solve the underlying function maximization problem. We identify two agent types: “naïve” (follows a random search strategy) and “skillful” (follows a Bayesian global optimization strategy). Through an extensive simulation study, we show that the assumption of the linear quality function is only valid for small effort levels. In general, the quality function is an increasing, concave function with derivative and curvature that depend on the problem complexity and agent’s skills.

Descriptive Models of Sequential Decisions in Engineering Design: An Experimental Study

“…10. The applications of this include design crowdsourcing where game-theoretic models lack design process models [2,40] and the agent-based models of engineering systems design where characterization of quality as a function of designer effort is difficult to achieve [3]. Furthermore, system engineers and managers can set the fixed budget at low values or provide monetary incentives for reducing spending to nudge a designer's decisions toward EU-based strategies, which are efficient for maximizing net payoff (design performance minus cost of evaluation).…”

“…This understanding can allow researchers to predict the outcomes of engineering design and systems engineering processes (e.g., Refs. [2,3]), to identify human-related sources of inefficiencies such as cognitive biases, and to find ways to reduce inefficiencies.…”

Modeling the System Acquisition Using Deep Reinforcement Learning

“…We define the quality function to be a stochastic process that maps the effort to a measurable outcome of a system that we call quality. In our previous work [7,23], we argued that a design problem can be modeled as an optimization problem where the goal is to maximize some attribute function and that the designer's search strategy follows certain heuristics that resemble Bayesian global optimization (BGO). The complexity of the system is specified with the smoothness of the underlying attribute function that describes the design problem.…”

“…At the largest time scale one considers the acquisition process as series of actions which are, request for bids, bidding and auctioning, contracting, and finally building and deploying the system, without resolving the fine details that occur within each step. At finer time scales, one may study different stages of the acquisition process from the intricate details of the entire systems engineering process [3,4] to communication between design teams [5] to how individual designers solve problems [6,7]. In this paper, we focus on the largest time scale, i.e., at the entire acquisition process.…”

Toward a Theory of Systems Engineering Processes: A Principal–Agent Model of a One-Shot, Shallow Process

“…The quality function is affected by what the principal believes about the task complexity and the problem solving skills of the agent. Following our work [12], we model the design task as a maximization problem where the agent seeks the optimal solution. The principal expresses their prior beliefs about the task complexity by modeling the objective function as a random draw from a Gaussian process prior with a suitably selected covariance function.…”