Evidence of the presence of bias in subjective metrics

Aranda, Alejandrina M.; Dieste, Óscar; Juristo, Natália

doi:10.1145/2601248.2601291

Cited by 11 publications

(10 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This did not happen, so complexity does not appear a priori to be a moderator variable threatening the validity of the results. This does not necessarily mean that there is no Knowledge x Complexity interaction, as subjects' personal perceptions are not always reliable [23].…”

Section: Effect Of Apparent Complexity On Problemsmentioning

confidence: 99%

“…However, computer engineers have been reported in the literature (e.g. : [52]) and observed [23] to be overoptimistic, and thus it is more of an inconvenience (as it is a potential validity threat) than a surprise.…”

Section: Familiarity With the Domain Versus Knowledge Of The Domainmentioning

confidence: 99%

“…This contrasts with Hadar et al [5] and Kristensson et al [7] experimental studies, where the domain knowledge is operationalized using a questionnaire (i.e., participants' personal opinion about how much they know). We considered that operationalizing the domain knowledge construct by asking subjects what they know about the domain is a rather subjective approach and may pose validity threats to the results [22], [23]. Therefore, our paper experimentally studies the influence of domain knowledge on an elicitation technique that has not been much researched but is very often used (the interview) and operationalizes knowledge objectively instead of subjectively.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Effect of Domain Knowledge on Elicitation Effectiveness: An Internally Replicated Controlled Experiment

Aranda

Dieste

Juristo

2016

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Abstract-Context. Requirements elicitation is a highly communicative activity in which human interactions play a critical role. A number of analyst characteristics or skills may influence elicitation process effectiveness. Aim. Study the influence of analyst problem domain knowledge on elicitation effectiveness. Method. We executed a controlled experiment with post-graduate students. The experimental task was to elicit requirements using open interview and consolidate the elicited information immediately afterwards. We used four different problem domains about which students had different levels of knowledge. Two tasks were used in the experiment, whereas the other two were used in an internal replication of the experiment; that is, we repeated the experiment with the same subjects but with different domains. Results. Analyst problem domain knowledge has a small but statistically significant effect on the effectiveness of the requirements elicitation activity. The interviewee has a big positive and significant influence, as does general training in requirements activities and interview experience. Conclusion. During early contacts with the customer, a key factor is the interviewee; however, training in tasks related to requirements elicitation and knowledge of the problem domain helps requirements analysts to be more effective. 1INTRODUCTIONR EQUIREMENTS elicitation, that is, seeking, capturing and consolidating requirements, is a core activity of any requirements engineering process [1] and has a direct influence on software quality [2]. Requirements elicitation depends on intensive communication between users and analysts in order to gather the right information [3]. Human interactions play an important role in this context. On one hand, customers should be able to interact and communicate their needs to analysts. On the other hand, analysts should be able to draw out and grasp the necessary domain information from customers.The effectiveness of requirements engineering activities is believed to partially depend on the participating individuals [4]. It has been observed that interview effectiveness can vary significantly depending on interviewer skills, probably because proficiency affects the course of the questioning [5]. As a result, elicitation strongly depends on the individual doing the interviewing [6]. Similar effects have been identified in brain-storming [4] and using other elicitation techniques [7].Several personal attributes may have a bearing on the effectiveness of any requirements-related task: experience [8]

show abstract

Section: Effect Of Apparent Complexity On Problemsmentioning

confidence: 99%

Section: Familiarity With the Domain Versus Knowledge Of The Domainmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Effect of Domain Knowledge on Elicitation Effectiveness: An Internally Replicated Controlled Experiment

Aranda

Dieste

Juristo

2016

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Several studies show that UX can be measured by either subjective methods or objective methods [11,16]. Yet objective measures tend to be more reliable and subjective measures must be well defined to avoid bias [1]. However, they may both bring different understanding of UX.…”

Section: Introductionmentioning

confidence: 99%

Effects on User Experience in an Edutainment Virtual Environment

Tcha-Tokey

Loup-Escande

Christmann

et al. 2017

Proceedings of the European Conference on Cognitive Ergonomics 2017

View full text Add to dashboard Cite

ABSTRACT1The user experience in immersive virtual environment can be broadly defined by the feeling of presence and immersion in a virtual environment. The propensity of feeling "there" and "enveloped by" in the virtual environment seems to be essential in offering a high quality of experience. However, this user experience definition is partial and neglects other components of the user experience in immersive virtual environments. We consider the overall user experience in immersive virtual environments. In this study, we use our unified questionnaire to measure, analyze and compare the user experience of two immersive virtual devices (i.e., a Cave Automatic Virtual Environment and a Mobile Head Mounted Display). 21 participants took place in this study and assessed the user experience of these devices through the "King Tut VR2" edutainment virtual application. The results show that the CAVE induced a greater user experience than the HMD with significant difference in presence, engagement, flow, skill, judgement and experience consequence. The results also show no significant difference in immersion, usability, emotion and technology adoption between the CAVE and the HMD.

show abstract

“…Hemos observado previamente que existen biases cuando los datos demográficos se miden en escalas de Likert (A. Aranda, Dieste, & Juristo, 2014) ya que los sujetos tienden a sobre valorar su experiencia verdadera.…”

Section: Cambios Del Experimento E-2015 Respecto a E-2013unclassified

Estudio empírico de la influencia de la experiencia y del conocimiento del dominio del analista en la efectividad de la educción de requisitos

King¹

View full text Add to dashboard Cite

Evidence of the presence of bias in subjective metrics

Cited by 11 publications

References 6 publications

Effect of Domain Knowledge on Elicitation Effectiveness: An Internally Replicated Controlled Experiment

Effect of Domain Knowledge on Elicitation Effectiveness: An Internally Replicated Controlled Experiment

Effects on User Experience in an Edutainment Virtual Environment

Estudio empírico de la influencia de la experiencia y del conocimiento del dominio del analista en la efectividad de la educción de requisitos

Contact Info

Product

Resources

About