Using Surveys and Web-Scraping to Select Tools for Software Testing Consultancy

Practitioner Evaluations on Software Testing Tools

“…Agreements on the criteria are important for confirming assumptions, but the disagreements (i.e., low values and outliers) are valuable for identifying possibly problematic issues. For example, low evaluations for criteria considered important in tool selection [44] are worth studying, in more detail. Our findings suggest that collective opinion can be used to point out issues, worth focusing on or investigating, in more detail.…”

“…Such sources may be useful for identifying tools, but the contents are neither generalizable nor validated for tool selection. There are software testing related academic studies which rely on surveys as the key methodology, e.g., [8,11,13,16,24,32,41,44], but only a few report software test tools (used by the practitioners) by name (e.g., [8,16,17]).…”

“…To evaluate software testing tools, we need collective information, knowledge from people having invested time in choosing and using tools [44]. Kitchenham et al [26] define a survey as a "comprehensive system for collecting information to describe, compare or explain knowledge, attitudes and behavior", and representativeness of responses can be justified by analyzing the demographics of the respondents [26].…”

“…We constructed a survey questionnaire including questions about background information and 15 questions for evaluating criteria (on different selected tools of choice), see Table 1. We used the criteria for the survey from a set of characteristics considered important by practitioners in test tool selection [44,45] and resting on the ISO/IEC 25010 1 quality model. The criteria to be evaluated were: (1) Applicability (2) Compatibility (3) Configurability (4) Cost-Effectiveness (5) Costs (6) Cross-Platform Support (7) Easy to Deploy (8) Easy to Use (9) Expandability (10) The respondents were able to select one or more tools and evaluate the criteria of choice for each tool, one tool at a time.…”

“…The criteria to be evaluated were: (1) Applicability (2) Compatibility (3) Configurability (4) Cost-Effectiveness (5) Costs (6) Cross-Platform Support (7) Easy to Deploy (8) Easy to Use (9) Expandability (10) The respondents were able to select one or more tools and evaluate the criteria of choice for each tool, one tool at a time. The list of tools (100) was created from a set of tools identified by practitioners for software testing [44]. The respondents could indicate the basis of their evaluations for the tool(s), i.e., whether those were based on personal experience using the tool, or on a generic opinion, e.g., from observing others using the tool.…”