Experiments, quasi-experiments, and case studies: A review of empirical methods for evaluating decision support systems

Adelman, Leonard

doi:10.1109/21.87078

Cited by 87 publications

(51 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…creativity (Snider, Culley, & Dekoninck, 2013), design review (Murphy, Ivarsson, & Lymer, 2012), and problem solving (Dorst & Cross, 2001;McDonnell, 2012). Here, micro--scale studies typically establish external validity (Adelman, 1991;Gray & Salzman, 1998) by building links to the wider design processes through, logical argument, theory building or testing, and explanatory models or frameworks. Less commonly this can be achieved through independent validation via integration with theoretically cohesive macro and meso--scale research.…”

Section: Micro--scalementioning

confidence: 99%

Activity Theory as a means for multi-scale analysis of the engineering design process: A protocol study of design in practice

2015

View full text Add to dashboard Cite

Section: Micro--scalementioning

confidence: 99%

Activity Theory as a means for multi-scale analysis of the engineering design process: A protocol study of design in practice

2015

View full text Add to dashboard Cite

“…Note that performance evaluation may use -besides simulation -analytical models, case studies, and real-life experiments; see the review of empirical techniques for evaluating IT systems by Adelman 48 and the taxonomic reviews of SCM research by Beamon, 6 Ganeshan et al, 49 and Otto and Kotzab. 30 …”

Section: More On Bsc and Simulationmentioning

confidence: 99%

Performance metrics in supply chain management

Kleijnen

Smits

2003

Journal of the Operational Research Society

276

157

View full text Add to dashboard Cite

This survey paper starts with a critical analysis of various performance metrics for supply chain management (SCM), used by a specific manufacturing company. Then it summarizes how economic theory treats multiple performance metrics. Actually, the paper proposes to deal with multiple metrics in SCM via the balanced scorecard -which measures customers, internal processes, innovations, and finance. To forecast how the values of these metrics will change -once a supply chain is redesigned -simulation may be used. This paper distinguishes four simulation types for SCM: (i) spreadsheet simulation, (ii) system dynamics, (iii) discrete-event simulation, and (iv) business games. These simulation types may explain the bullwhip effect, predict fill rate values, and educate and train users. Validation of simulation models requires sensitivity analysis; a statistical methodology is proposed. The paper concludes with suggestions for a possible research agenda in SCM. A list with 50 references for further study is included.

show abstract

“…In addition, an important experimental design consideration for human subject testing is external validity, which is a measure of how well experimental results will generalize to an operational setting. One way to strengthen external validity is to represent the operational environment as accurately as possible in an experimental setting (Adelman, 1991).…”

Section: Introductionmentioning

confidence: 99%

The Need for Command and Control Instant Message Adaptive Interfaces: Lessons Learned from Tactical Tomahawk Human-in-the-Loop Simulations

Cummings

2004

CyberPsychology & Behavior

View full text Add to dashboard Cite

In the recent development of a human-in-the-loop simulation test bed designed to examine human performance issues for supervisory control of the Navy's new Tactical Tomahawk missile, measurements of operator situation awareness (SA) and workload through secondary tasking were taken through an embedded instant messaging program. Instant message interfaces (otherwise known as "chat"), already a means of communication between Navy ships, allow researchers to query users in real-time in a natural, ecologic setting, and thus provide more realistic and unobtrusive measurements. However, in the course of this testing, results revealed that some subjects fixated on the real-time instant messaging secondary task instead of the primary task of missile control, leading to the overall degradation of mission performance as well as a loss of SA. While this research effort was the first to quantify command and control performance degradation as a result of instant messaging, the military has recognized that in its network centric warfare quest, instant messaging is a critical informal communication tool, but has associated problems. Recently a military spokesman said that managing chat in current military operations was sometimes a "nightmare" because military personnel have difficulty in handling large amounts of information through chat, and then synthesizing knowledge from this information. This research highlights the need for further investigation of the role of instant messaging interfaces both on task performance and situation awareness, and specifies how the associated problems could be ameliorated through adaptive display design.

show abstract

Experiments, quasi-experiments, and case studies: A review of empirical methods for evaluating decision support systems

Cited by 87 publications

References 13 publications

Activity Theory as a means for multi-scale analysis of the engineering design process: A protocol study of design in practice

Activity Theory as a means for multi-scale analysis of the engineering design process: A protocol study of design in practice

Performance metrics in supply chain management

The Need for Command and Control Instant Message Adaptive Interfaces: Lessons Learned from Tactical Tomahawk Human-in-the-Loop Simulations

Contact Info

Product

Resources

About