On the Use of Experiments in Design Science Research: A Proposition of an Evaluation Framework

Mettler, Tobias; Eurich, Markus; Winter, Robert

doi:10.17705/1cais.03410

Cited by 36 publications

(28 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While these articles label themselves as SLR, more than half of them do not define what SLRs entail. Instead, Grahlmann et al (2012), Gräning et al (2011), Guillemette and Paré (2012), Hummel et al (2013) and Mettler et al (2014) all refer to Webster and Watson (2002) as presumably offering SLR guidelines (which they are not). Furthermore, studies that do describe SLRs are often vague in their description, with only one SLR including the use of a protocol, arguably the most central part for a SLR.…”

Section: Criteria Referencesmentioning

confidence: 99%

On being ‘Systematic’ in Literature Reviews in IS

Boell

Ćećez-Kecmanović

2015

Journal of Information Technology

307

218

View full text Add to dashboard Cite

General guidelines for conducting literature reviews often do not address the question of literature searches and dealing with a potentially large number of identified sources. These issues are specifically addressed by so-called systematic literature reviews (SLRs) that propose a strict protocol for the search and appraisal of literature. Moreover, SLRs are claimed to be a 'standardized method' for literature reviews that is replicable, transparent, objective, unbiased and rigorous, and thus superior to other approaches for conducting literature reviews. These are significant and consequential claims that -despite increasing adoption of SLRs -remained largely unnoticed in the information systems (IS) literature. The objective of this debate is to draw attention of the IS community to SLR's claims, to question their justification and reveal potential risks of their adoption. This is achieved by first examining the origins of SLR and the prescribed SLR process and then by critically assessing their claims and implications. In this debate, we show that SLRs are applicable and useful for a very specific kind of literature review, a meta study that identifies and summarizes evidence from earlier research. We also demonstrate that the claims that SLRs provide superior quality are not justified. More importantly, we argue that SLR as a general approach to conducting literature reviews is highly questionable, concealing significant perils. The paper cautions that SLR could undermine critical engagement with literature and what it means to be scholarly in academic work.

show abstract

Section: Criteria Referencesmentioning

confidence: 99%

On being ‘Systematic’ in Literature Reviews in IS

Boell

Ćećez-Kecmanović

2015

Journal of Information Technology

307

218

View full text Add to dashboard Cite

show abstract

“…Optimally, a set of genuine conversations would be available, however, genuine homeowners and police's security department did not agree on fully fledged recording of the highly sensitive information. Consequently, we use data collected during a design experiment (Mettler et al, 2014) where recording of the conversations was accepted. The experiment was designed to collect user assessment of the SmartProtector's value for persuading the advisee (Dolata et al, 2016) and to observe SmartProtector's impact on vocabulary.…”

Section: Data Collectionmentioning

confidence: 99%

Translation and Adoption: Exploring Vocabulary Work in Expert-Layperson Encounters

Dolata

Schwabe

2019

Comput Supported Coop Work

View full text Add to dashboard Cite

An advisory service encounter brings together a domain expert with a layperson in a complex life situation. Because of the different backgrounds and expertise levels, the interlocutors and meanings is an essential part of advisory services and, generally, of expert-layperson collaboration. Establishing and maintaining a common lexicon is a specific and, at the same type, frequent type of collaborative work. Nevertheless, it remains unclear what efforts this collaborative work involves and what role collaborative IT applications play in this regard. A collaborative application can well support the maintenance of a common lexicon by providing a way to externalize terms or definitions. Or it can generate additional work by providing further terms and definitions to be incorporated in the common lexicon. That puzzle gets reflected in specific design dilemmas: should the system use expert or conventional terms, what is the source of the adequate terminology, to what extent should the system adapt to the individual lexical choices, etc. This manuscript explores the work involved in establishing and maintaining a common lexicon in advisory services between an expert and a layperson. In particular, it demonstrates how external material, a dedicated collaborative application developed for supporting advisory services, impacts the maintenance of a common lexicon. First, the manuscript depicts practices involved in translation and adoption of terminology from the system into the conversation. Second, it characterizes the system's impact on interlocutors' vocabulary. Overall, the study contributes to the discourse on expert-layperson collaboration by characterizing an important type of work, the vocabulary work, and by depicting the role of collaborative applications for this type of work.Abstract. An advisory service encounter brings together a domain expert with a layperson in a complex life situation. Because of the different backgrounds and expertise levels, the interlocutors require a common lexicon to guarantee for smooth collaboration. Consequently, the negotiation of terms and meanings is an essential part of advisory services and, generally, of expert-layperson collaboration. Establishing and maintaining a common lexicon is a specific and, at the same type, frequent type of collaborative work. Nevertheless, it remains unclear what efforts this collaborative work involves and what role collaborative IT applications play in this regard. A collaborative application can well support the maintenance of a common lexicon by providing a way to externalize terms or definitions. Or it can generate additional work by providing further terms and definitions to be incorporated in the common lexicon. That puzzle gets reflected in specific design dilemmas: should the system use expert or conventional terms, what is the source of the adequate terminology, to what extent should the system adapt to the individual lexical choices, etc. This manuscript explores the work involved in establishing and maintaining a common lexicon in advis...

show abstract

“…Experiments that test the effects of different conditions by adhering to the principles of control, randomization, and manipulation (Mettler et al, 2014) Theory contribution (Gregor, 2006) Analyzing Says what is. The theory does not extend beyond analysis and description.…”

Section: Simulation Typementioning

confidence: 99%

“…In these studies, groups of people participate in role-playing in which they interact in a simulated world (e.g., business game). HU simulations are also referred to as experiments that test the effects of different conditions by adhering to the principles of control, randomization, and manipulation (Mettler et al, 2014). The use of HU simulation is common when the behavior of an agent becomes too complex for one to embed it into the artificial agent's cognitive model.…”

Section: Adaptationmentioning

confidence: 99%

Exploring Foundations for Using Simulations in IS Research

Za¹,

Spagnoletti²,

Winter³

et al. 2018

CAIS

Self Cite

View full text Add to dashboard Cite

Researchers in many fields have adopted simulation to understand a system's behavior by imitating it through an artificial object that exhibits nearly identical behavior. Although simulation approaches have been widely adopted for theory building in fields such as engineering, computer science, management, and social sciences, researchers in the IS field often overlook their potential. In this paper, we examine how IS research uses different simulation approaches and, thereby, provide insights and methodological recommendations for future studies. From reviewing the literature on simulation studies published in top-tier IS journals, we define three classes of simulations: the self-organizing, the elementary, and the situated. We identify a set of stylized facts for characterizing the ways in which IS simulation studies present the premise, the inference, and the contribution. As a result, we provide guidance to future simulation researchers in designing and presenting their findings.

show abstract

On the Use of Experiments in Design Science Research: A Proposition of an Evaluation Framework

Cited by 36 publications

References 51 publications

On being ‘Systematic’ in Literature Reviews in IS

On being ‘Systematic’ in Literature Reviews in IS

Translation and Adoption: Exploring Vocabulary Work in Expert-Layperson Encounters

Exploring Foundations for Using Simulations in IS Research

Contact Info

Product

Resources

About