Evaluating the complexity of engineered systems: A framework informed by a user case study

Abstract

Evaluating the complexity of an engineered system is challenging for any organization, even more so when operating in a System‐of‐Systems (SoS) context. Here, we analyze one particular decision support tool as an illustratory case study. This tool has been used for several years by Thales Group to evaluate system complexity across a variety of industrial engineering projects. The case study is informed by analysis of semistructured interviews with systems engineering experts within the Thales Group. This analysis reveals a number of positive and negative aspects of (i) the tool itself and (ii) the way in which the tool is embedded operationally within the wider organization. While the first set of issues may be solved by making improvements to the tool itself, informed by further comparative analysis and growing literature on complexity evaluation, the second “embedding challenge” is distinct, seemingly receiving less attention in the literature. In this paper, we focus on addressing this embedding challenge, by introducing a complexity evaluation framework, designed according to a set of principles derived from the case study analysis; namely that any effective complexity evaluation activity should feature collaborative effort toward building an evaluation informed by a shared understanding of contextually relevant complexity factors, iterative (re‐)evaluation over the course of a project, and progressive refinement of the complexity evaluation tools and processes themselves through linking project evaluations to project outcomes via a wider organizational learning cycle. The paper concludes by considering next steps including the challenge of assuring that such a framework is being implemented effectively.

Toward an integrated project complexity narrative – A case study of academic organizations

“…The remaining significant challenges towards addressing the current research gaps identified by Shenhar, et al 3 . and Potts, et al 16 . are the lack of any single, agreed definitions of system complexity 11 and project complexity 5 …”

“…We subscribe to an understanding of complexity as discussed by Sheard and Mostashari, 17,24 that is, complexity can be viewed as the inability to predict the behavior of a system due to large numbers of constituent parts within the system and dense relationships among them 95 . In complex projects and systems, these constituent parts may be technical, economic, social and cultural in nature and contribute toward structural, dynamic, and social‐political complexity 16,22,54 . The large body of literature addressing structural, dynamics and socio‐political complexities motivated us to follow Geraldi, et al 22 .…”

“…This implies the need to embrace a broader research agenda for SE, including how we introduce SE to the future engineers 35,44 . By introducing a project complexity narrative that supports sense‐making as a collaborative and iterative process one can sidesteps the challenges associated with the lack of any single, agreed definition of system complexity 11,16 …”

“…This aligns with the three types of structural complexity described by Sheard and Mostashari 17 that is, size, architecture and connectivity. It is also the type of complexity that has seen the most extensive development of complexity metrics 16,68,69 and the concept with the most mentions in both project and system complexity literature. Structural complexity should be understood as applicable to both engineered systems and the organizations put in place to deliver them.…”

“…[11][12][13][14] Watson et al 15 describe and define distinguishing characteristics that can be used to differentiate between complex and non-complex systems and discuss how complexity can be managed in light of these characteristics. Potts et al 16 provide a review of literature related to challenges in system complexity evaluations and discuss challenges involved in operationally embedding complexity evaluations within an organization. For example, does the organization evaluate the technical system to be developed, the project to realize the technical system or both?…”