Päivi Raulamo-Jurvanen M3S, ITEE, Test automation is important in the software industry but selfassessment instruments for assessing its maturity are not sufficient. The two objectives of this study are to synthesize what an organization should focus to assess its test automation; develop a self-assessment instrument (a survey) for assessing test automation maturity and scientifically evaluate it. We carried out the study in four stages. First, a literature review of 25 sources was conducted. Second, the initial instrument was developed. Third, seven experts from five companies evaluated the initial instrument. Content Validity Index and Cognitive Interview methods were used. Fourth, we revised the developed instrument. Our contributions are as follows: (a) we collected practices mapped into 15 key areas that indicate where an organization should focus to assess its test automation; (b) we developed and evaluated a self-assessment instrument for assessing test automation maturity; (c) we discuss important topics such as response bias that threatens self-assessment instruments. Our results help companies and researchers to understand and improve test automation practices and processes. CCS CONCEPTS• Software and its engineering → Software testing and debugging; Software development process management. Some test maturity models provide self-assessment instruments for organizations. For example, TPI provides a Test Maturity Matrix [34], TMap provides Checklists [1], TOM provides Test Organization Maturity Questionnaire [36]. The assessment items were defined in the form of survey questions. The assessment procedure is rather simple. Organizations answer assessment items (survey questions) to assess the maturity state of their test process [1,34,36].

No abstract

No abstract

No abstract

In software engineering practice, evaluating and selecting the software testing tools that best fit the project at hand is an important and challenging task. In scientific studies of software engineering, practitioner evaluations and beliefs have recently gained interest, and some studies suggest that practitioners find beliefs of peers more credible than empirical evidence. To study how software practitioners evaluate testing tools, we applied online opinion surveys (n=89). We analyzed the reliability of the opinions utilizing Krippendorff's alpha, intra-class correlation coefficient (ICC), and coefficients of variation (CV). Negative binomial regression was used to evaluate the effect of demographics. We find that opinions towards a specific tool can be conflicting. We show how increasing the number of respondents improves the reliability of the estimates measured with ICC. Our results indicate that on average, opinions from seven experts provide a moderate level of reliability. From demographics, we find that technical seniority leads to more negative evaluations. To improve the understanding, robustness, and impact of the findings, we need to conduct further studies by utilizing diverse sources and complementary methods.