Empirical analysis of factors affecting confirmation bias levels of software engineers

Abstract

Confirmation bias is defined as the tendency of people to seek evidence that verifies a hypothesis rather than seeking evidence to falsify it. Due to the confirmation bias, defects may be introduced in a software product during requirements analysis, design, implementation and/or testing phases. For instance, testers may exhibit confirmatory behaviour in the form of a tendency to make the code run rather than employing a strategic approach to make it fail. As a result, most of the defects that have been introduced in the earlier phases of software development may be overlooked leading to an increase in software defect density. In this paper, we quantify confirmation bias levels in terms of a single derived metric. However, the main focus of this paper is the analysis of factors affecting confirmation bias levels of software engineers. Identification of these factors can guide project managers to circumvent negative effects of confirmation bias; as well as providing guidance for the recruitment and effective allocation of software engineers. In this empirical study, we observed low confirmation bias levels among participants with logical reasoning and hypothesis testing skills.

A controlled experiment on time pressure and confirmation bias in functional software testing

“…Low confirmation bias levels were observed in participants who were experienced but inactive in development or testing . No effect of company size, educational background (undergraduate) or educational level (bachelor's, master's) were observed (Calikli and Bener 2014). However, the job title (researcher) was associated with significantly lower levels of confirmation bias when compared to the rest of the job titles, which is possibly due to researchers' acquired critical and analytical skills (Calikli and Bener 2014).…”

“…Later, studied the effects of experience and activeness (in development and testing), along with the logical reasoning skills gained through education, on confirmation bias. In 2014, they extended their set of factors to include job title (developer, tester, analyst, and researcher), education, development methods and company size (small and medium enterprises [SME] vs large-scale companies) (Calikli and Bener 2014). Both studies showed no effect of either development methods or experience in development and testing on confirmation bias, but as an effect of logical reasoning skills, low bias levels were observed Bener 2010, 2014).…”

“…Human cognition is related to human thought processes. When a thought process is based on illogical reasoning, it might negatively impact the performed task (Calikli and Bener 2014). Formally, this phenomenon is known as cognitive bias and is defined as "...cognitions or mental behaviours that prejudice decision quality in a significant number of decisions for a significant number of people" (Arnott 2006, p. 59).…”

“…In this paper, we limit the scope of the investigation to functional software testing. The manifestation of confirmation bias among software testers is considered based on their inclination towards designing test cases that confirm the correct functioning of an application rather than scenarios that break the code (Calikli and Bener 2014;Teasley et al 1994;Salman 2016). In this regard, limited empirical evidence is available in the SE literature that reports confirmation bias manifestation in software testing resulting in deteriorated software quality, e.g.…”

Personality, confirmation bias, and forensic interviewing performance

“…Confirmation bias is a universal trait in human cognition (Kahneman & Tversky, 1982), and is detected in many different professionals, for example, in software programmers (Calikli & Bener, 2014). Such biases are relevant in the legal setting because they will tend to lead to poor decisions at every stage in the legal system.…”