Survey Research in Software Engineering: Problems and Mitigation Strategies

Abstract

Background: The need for empirical investigations in software engineering is growing. Many researchers nowadays, conduct and validate their solutions using empirical research. Survey is one empirical method which enables researchers to collect data from a large population. Main aim of the survey is to generalize the findings. Aims: In this study we aim to identify the problems researchers face during survey design, and mitigation strategies. Method: A literature review as well as semi-structured interviews with nine software engineering researchers were conducted to elicit their views on problems and mitigation strategies. The researchers are all focused on empirical software engineering. Results: We identified 24 problems and 65 strategies, structured according to the survey research process. The most commonly discussed problem was sampling, in particular the ability to obtain a sufficiently large sample. To improve survey instrument design, evaluation and execution recommendations for question formulation and survey pre-testing were given. The importance of involving multiple researchers in the analysis of survey results was stressed. Conclusions: The elicited problems and strategies may serve researchers during the design of their studies. However, it was observed that some strategies were conflicting. This shows that it is important to conduct a trade-off analysis between strategies.

Prioritizing challenges of agile process in distributed software development environment using analytic hierarchy process

“…On the basis of the findings of the literature review, an online survey questionnaire to identify the challenges in distributed agile development was developed. According to Khan et al, a questionnaire‐based empirical study is a suitable method for collecting self‐reporting qualitative and quantitative data from experts using techniques such as questionnaires and interviews . In the present study, the questionnaire was divided into 5 sections.…”

Are we teaching UML according to what IT companies need? A survey on the São Carlos-SP region

“…We chose a survey as a methodology for this study because of its benefits such as the possibility to remotely collect data, the time-frame independence for data collection, and the ease of distribution [10]. We tried to mitigate problems with surveys by following the recommendations of Ghazi et al [14] and Linåker et al [17].…”

Using an Integrated Framework for Conceptual Modeling

“…The categorical data were analyzed using SPSS in order to discover if there was a relationship between categorical variables. In this case, were used the Fisher`s Exact test because of small sample size [22][23][24][25]. The results of this test are presented in the section 4.…”

“…It depends on the aim of the study, sample specificity, use of established theory, quality of dialogue, and analysis strategy [24]. Furthermore, the authors Ghazi, A. N., Petersen, K., Reddy, S. S. V. R., & Nekkanti, H. (2017) based on literature review highlighted that this is a problem in the field of software engineering [25]. In the following sections, the outcomes of the Mixed Method Case Studies should be presented.…”