Qualitative methods for engineering systems: Why we need them and how to use them

Self Cite

Controlled experiments ranging from laboratory experiments to field experiments are becoming increasingly important within systems engineering and design (SE&D) research. Controlled experiments provide unique capabilities for testing hypotheses, and guiding the development of scientific theories, but their potential is not being fully exploited. As the systems engineering research community embraces different research methods commonly adopted in other domains, there is a need for understanding the role and appropriateness of those methods within the research process. While the design of experiments in SE&D research is not unique per se, we contend that aspects of representativeness in the SE&D context pose different experimental design challenges than do the problems typically studied by experimental social scientists. There is therefore a need to translate established practices to the SE&D context, and also belay widely held concerns about the value of experiments in SE&D research. To address this need, the goals of the paper are to discuss the potential role of controlled experiments in systems engineering and design research, to review the benefits and limitations of different types of experiments, and to highlight the key issues in validating experimental studies. The taxonomy and the discussion on validation build on the rich history in social and behavioral sciences, with specific adaptation to the SE&D context. The discussion is anchored in specific examples from the authors’ own research on open innovation in systems engineering.

Section: Natural Field Experiments Can Balance Contextual Richness Wimentioning

confidence: 97%

Section: Examples Of Experiments In Systems Engineering and Design (Smentioning

confidence: 99%

Experiments in systems engineering and design research

Panchal

Szajnfarber

2017

Self Cite

“…In order to gain an in‐depth understanding of how oversight impacts engineering work at the contractor level, we used qualitative research methods . Since it has been suggested that previously measured categories of oversight‐related activities are incomplete, there is a need to inductively develop a comprehensive perspective of the categories of work performed for oversight‐related purposes .…”

Section: Research Approachmentioning

confidence: 99%

How government oversight adds time to contractor engineering work

Brainard

Szajnfarber

2018

Self Cite

Oversight is necessary for evaluating the performance of contractors building complex systems for the U.S. government, but it can also result in additional program costs. In order to accurately measure the extent of these additional costs, stakeholders need to understand how oversightrequired activities add time to work. To that end, this paper presents the data from a qualitative research study to investigate how oversight adds time to contractor engineering work at a major U.S. aerospace company. We observed that oversight activities require many tasks to complete, which can add cascades of new activities that are performed in multiple iterations. Oversight activities cannot be accounted for in a binary way-oversight adds time to parts of all activities that the contractor performs. We describe the mechanisms through which oversight impacts work by extending the time spent on all activities. Understanding how added time occurs could help both contractors and government program managers determine ways to reduce unnecessary costs. Implications of these findings for systems engineers interested in conducting studies to measure the impact of oversight are discussed. K E Y W O R D Sacquisition, oversight 54

“…Thus, qualitative methods aim to produce theories whereas quantitative methods aim to test theories’ predictions. An excellent overview of qualitative methods is given in this special issue …”

Section: Relationship To Other Standards Of Rigormentioning

confidence: 99%

“…An excellent overview of qualitative methods is given in this special issue. 93 Not all hypothesis generation techniques are qualitative. Indeed, from the perspective of predictive theorizing, axiomatic methods also constitute a source of causal theories that can be tested to determine if they correspond to empirical findings.…”

Section: Relationship To Hypothesis Generation Techniquesmentioning

confidence: 99%

Assessing causal claims about complex engineered systems with quantitative data: internal, external, and construct validity

Broniatowski

Tucker

2017

Engineers seek to design systems that will produce an intended change in the state of the world. How are we to know if a system will behave as intended? This article addresses ways that this question can be answered. Specifically, we focus on three types of research validity: (1) internal validity, or whether an observed association between two variables can be attributed to a causal link between them; (2) external validity, or whether a causal link generalizes across contexts; and (3) construct validity, or whether a specific set of metrics corresponds to what they are intended to measure. In each case, we discuss techniques that may be used to establish the corresponding type of validity: namely, quasi‐experimental design, replication, and establishment of convergent‐discriminant validity and reliability. These techniques typically require access to data, which has historically been limited for research on complex engineered systems. This is likely to change in the era of “big data.” Thus, we discuss the continued utility of these validity concepts in the face of advances in machine learning and big data as they pertain to complex engineered sociotechnical systems. Next, we discuss relationships between these validity concepts and other prominent approaches to evaluating research in the field. Finally, we propose a set of criteria by which one may evaluate research utilizing quantitative observation to test causal theory in the field of complex engineered systems.