Design method validation – an investigation of the current practice in design research

Eisenmann, Matthias; Grauberger, Patric; Üreten, Selin; Krause, Dieter; Matthiesen, Sven

doi:10.1080/09544828.2021.1950655

Cited by 17 publications

(12 citation statements)

References 87 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A more recent analysis of studies aiming at design method validation reports a lack of success evaluation. A possible explanation given by the authors is that aside from the high effort required by such studies strategies for the transition of methods to practice are missing (Eisenmann et al 2021).…”

Section: Validation Of Design Methodsmentioning

confidence: 99%

Elements of a design method – a basis for describing and evaluating design methods

2022

View full text Add to dashboard Cite

Method development is at the heart of design research as methods are a formalised way to express knowledge about how aspects of design could or should be done. However, assuring that methods are in fact used in industry has remained a challenge. Industry will only use methods that they can understand and that they feel will give them benefit reliably. To understand the challenges involved in adopting a method, the method needs to be seen in context: it does not exist in isolation but forms a part of an ecosystem of methods for tackling related design problems. A method depends on the knowledge and skills of the practitioners using it: while a description of a method is an artefact that is a formalisation of engineering knowledge, a method in use constitutes a socio-technical system depending on the interaction of human participants with each other as well as with the description of the method, representations of design information and, often, tools for carrying out the method’s tasks. This paper argues that crucial factors in the adoption of methods include how well they are described and how convincingly they are evaluated. The description of a method should cover its core idea, the representations in which design information is described, the procedure to be followed, its intended use, and the tools it uses. The account of a method’s intended use should cover its purpose, the situations or product types within its scope, its coverage of kinds of problems within its scope, its expected benefit and conditions for its use. The different elements need to be evaluated separately as well as the method as an integrated whole. While verification and validation are important for some elements of methods, it is rarely possible to prove the validity of a method. Rather the developers of methods need to gather sufficient evidence that a method will work within a clearly articulated scope. Most design methods do not have binary success criteria, and their usefulness in practice depends as much on simplicity and usability as on the outcomes they produce. Evaluation should focus on how well they work, and how they can be customised and improved.

show abstract

Section: Validation Of Design Methodsmentioning

confidence: 99%

Elements of a design method – a basis for describing and evaluating design methods

2022

View full text Add to dashboard Cite

show abstract

“…However, studies show that this is not sufficiently considered or even not considered at all (Barth et al, 2011;Cantamessa, 2003). The biggest challenges when validating design methods involve the human as an object of observation, as well as the volatility of possible product design environments and problems (Eisenmann et al, 2021). This complexity and uniqueness of each design situation complicate the repeatability of validation activities for design methods (Reich, 2010).…”

Section: Validation Of Design Methodsmentioning

confidence: 99%

“…As a result, an objective evaluation of the benefits and efforts of a method application is only possible to a limited extent (Pedersen et al, 2000;Zanker, 1999). Moreover, method validation often focuses solely on investigating and evaluating applicability (Eisenmann et al, 2021). Nevertheless, some approaches can be found in the literature that deal with the validation of design methods to overcome the aforementioned challenges such as the Validation Square according to Pedersen et al (2000), the Concept Map according to Üreten et al (2019) as well as the Design Research Methodology (DRM) according to Blessing and Chakrabarti (2009).…”

Section: Validation Of Design Methodsmentioning

confidence: 99%

Validating a Design Method to Improve Collaboration in Distributed Product Design

Duehr¹,

Kopp²,

Rapp³

et al. 2022

How Product and Manufacturing Design Enable Sustainable Companies and Societies

View full text Add to dashboard Cite

One promising approach to deal with the increasingly complex and volatile conditions of product design is to work in distributed teams. However, this also poses new challenges that can result in efficiency and effectiveness losses. To overcome these challenges, the EDiT method -Enabling Distributed Teams -is currently being developed to enable distributed teams to identify and exploit improvement potentials in collaboration. To develop design methods that address the needs of their intended users, continuous and iterative validation during the whole method development process is of particular importance. This enables design methods to be adapted to the needs of their users at an early stage and to be further enhanced in a targetoriented manner. The question arises, what needs to be considered in supporting the continuous and iterative validation of the EDiT method. To address this need, this contribution focuses on the development of a process model that supports the method developer with the continuous and iterative validation of the EDiT method. To generate an understanding of what needs to be considered in supporting the validation of the EDiT method based on existing frameworks for method validation, three partial models were examined using a literature review, expert interviews, and a questionnaire. Each partial model yielded one key result: six paradigms outlining the validation of design methods, target relations to illustrate the interrelationships of the influences of the EDiT method, and nine requirements that are placed on a process model for the validation of the EDiT method. The underlying theoretical assumptions generated by the three partial models were eventually integrated into the process model. Finally, the results can be used to continuously improve the EDiT method while simultaneously demonstrate its benefits, to ultimately provide the potential for increased method acceptance in practice.

show abstract

“…Many scholarly efforts in relation to design methods have focused on the validation of individual methods, often in relation to specific lab-managed protocols or other means of controlling for design complexity, including both experimental and implementation-focused studies. Eisenmann et al [23] report on a recent survey of such design validation literature, describing issues of alignment that relate to a lack of shared vocabulary across methods (including relevant metrics to compare performance), describing this challenge as follows: "most design method developers set goals for their own methods and therefore develop a separate operationalisation resulting in a multitude of metrics. This makes it very challenging to compare methods with each other and hinders researchers to build a common standard for similar methods."…”

Section: Methods and Their Use By Practitionersmentioning

confidence: 99%

“…These kinds of actions would likely increase the practice-resonance of standalone methods and method discovery tools. However, this reorientation of method prescriptions would require a new commitment not only to method "validity" in the traditional performative sense [23], but also a consideration of method comparability, adaptability, and creativity as part of a broader ecosystem of tools, as well as practitioner competencies to translate these prescriptions to shape ethically-focused practice.…”

Section: Impacts Of Practice Resonance On the Design Of Ethics-focuse...mentioning

confidence: 99%

Scaffolding Ethics-Focused Methods for Practice Resonance

Gray¹,

Chivukula²,

Thomas³

et al. 2022

Preprint

View full text Add to dashboard Cite

Numerous methods and tools have been proposed to motivate or support ethical awareness in design practice. However, many existing resources are not easily discoverable by practitioners. One reason being that they are framed using language that is not immediately accessible or resonant with the felt complexity of everyday practice. In this paper, we propose a set of empirically-supported "intentions" to frame practitioners' selection of relevant ethics-focused methods based on interviews with practitioners from a range of technology and design professions, and then leverage these intentions in the design and iterative evaluation of a website that allows practitioners to identify supports for ethics-focused action in their work context. Building on these findings, we propose a set of heuristics to evaluate the practice resonance of resources to support ethics-focused practice, laying the groundwork for increased ecological resonance of ethics-focused methods and method selection tools. CCS Concepts: • Human-centered computing → Empirical studies in HCI; HCI design and evaluation methods; Usability testing.

show abstract

Design method validation – an investigation of the current practice in design research

Cited by 17 publications

References 87 publications

Elements of a design method – a basis for describing and evaluating design methods

Elements of a design method – a basis for describing and evaluating design methods

Validating a Design Method to Improve Collaboration in Distributed Product Design

Scaffolding Ethics-Focused Methods for Practice Resonance

Contact Info

Product

Resources

About