Practitioner Evaluations on Software Testing Tools

Abstract

In software engineering practice, evaluating and selecting the software testing tools that best fit the project at hand is an important and challenging task. In scientific studies of software engineering, practitioner evaluations and beliefs have recently gained interest, and some studies suggest that practitioners find beliefs of peers more credible than empirical evidence. To study how software practitioners evaluate testing tools, we applied online opinion surveys (n=89). We analyzed the reliability of the opinions utilizing Krippendorff's alpha, intra-class correlation coefficient (ICC), and coefficients of variation (CV). Negative binomial regression was used to evaluate the effect of demographics. We find that opinions towards a specific tool can be conflicting. We show how increasing the number of respondents improves the reliability of the estimates measured with ICC. Our results indicate that on average, opinions from seven experts provide a moderate level of reliability. From demographics, we find that technical seniority leads to more negative evaluations. To improve the understanding, robustness, and impact of the findings, we need to conduct further studies by utilizing diverse sources and complementary methods.

Applying Surveys and Interviews in Software Test Tool Evaluation

“…Rather, costs are categorized as a tool external factor [21]. In our prior research on tool evaluations [22], we found that practitioner evaluations for a tool, in a survey, may be dispersed. To improve understanding and robustness of the results, we analyze the topic using a complementary method.…”

Characterization of software testing practices: A replicated survey in Costa Rica

“…• To analyze what factors may influence the selection of software testing practices (Fernández-Sanz et al, 2009;Greiler et al, 2012;Pham et al, 2013;Pérez et al, 2013;Pfahl et al, 2014;Deak, 2014;Kochhar et al, 2015;Lima and Faria, 2016;Kochhar et al, 2019;Raulamo-Jurvanen et al, 2019). • To analyze software testing practices and the level of maturity in the industry (Fernández-Sanz, 2005;Grindal et al, 2006;Park et al, 2008).…”

“…Studies reported the gap between software testing state of the art and state of the practice (Ng et al, 2004;Dias-Neto et al, 2006;Sung and Paynter, 2006;Causevic et al, Software testing is still reported as a time consuming and expensive phase in software development (Beck and Perkins, 1983;Ng et al, 2004;Dias-Neto et al, 2006). The automation of software testing has continued its growth and there are opportunities for automated software testing research (Ghazi et al, 2015;Hynninen et al, 2018;Kochhar et al, 2019;Raulamo-Jurvanen et al, 2019).…”

“…Additionally, other studies surveying different aspects related to testing practices were conducted in Finland (Taipale et al, 2005(Taipale et al, , 2006Kasurinen et al, 2010;Pfahl et al, 2014;Smolander et al, 2016;Hynninen et al, 2018;Raulamo-Jurvanen et al, 2019), Spain (Fernández-Sanz, 2005Fernández-Sanz et al, 2009), Sweden (Runeson, 2006;Grindal et al, 2006;Engström and Runeson, 2010), Korea (Park et al, 2008;Yli-Huumo et al, 2014), Netherlands (Vonken et al, 2012), Norway , Belgium (Pérez et al, 2013), Turkey (Garousi et al, 2015, Sri Lanka (Vasanthapriyan, 2018), and Bangladesh (Bhuiyan et al, 2018). Finally, other studies surveying different aspects related to testing practices were conducted in different countries (Chan et al, 2005;Causevic et al, 2010;Rafi et al, 2012;Lee et al, 2012;Greiler et al, 2012;Pham et al, 2013;Daka and Fraser, 2014;Kanij et al, 2014;Deak, 2014;Ghazi et al, 2015;Kochhar et al, 2015;Lima and Faria, 2016;Rodrigues and Dias-Neto, 2016;Garousi et al, 2017;Kochhar et al, 2019).…”

Piloto de automatización de pruebas de software en la Unidad de Servicios Tecnológicos del Centro de Servicios y Gestión Empresarial (CESGE) del SENA

“…Además, es fundamental realizar una revisión de estrategias encaminadas a agilizar los procedimientos de pruebas funcionales iterativas [10], en donde toma relevancia el concepto de automatización para aminorar la intervención de las personas en las actividades repetitivas, reduciendo el tiempo y esfuerzo requerido y aumentando la cobertura de los casos de pruebas [11], [12]. Dentro de la implementación de estrategias de pruebas en las organizaciones es recomendable integrar el software de automatización como herramienta de apoyo y contribución a la efectividad y calidad; algunos framework son el TestNG y JUunit y dentro de estos, es habitual usar herramientas como SeleniumHQ, Watir, Capibara y Cucumber, las cuales básicamente automatizan los navegadores a través de tareas que los usuarios podrían hacer en aplicaciones web [13], [14], [15] [16]; así como también se encuentra Robot Framework con el cual es posible realizar pruebas de aceptación y desarrollo, [17] y CodeceptJS que es de tipo E2E cuya visión es fundamentada en ambientes y desarrollo por comportamiento (BDD). [8], [18].…”