Development of Scientific Software: A Systematic Mapping, a Bibliometrics Study, and a Paper Repository

Abstract

Scientific and engineering research is heavily dependent on effective development and use of software artifacts. Many of these artifacts are produced by the scientists themselves, rather than by trained software engineers. To address the challenges in this area, a research community often referred to as "Development of Scientific Software" has emerged in the last few decades. As this research area has matured, there has been a sharp increase in the number of papers and results made available, and it has thus become important to summarize and provide an overview about those studies. Through a systematic mapping and bibliometrics study, we have reviewed 130 papers in this area. We present the results of our study in this paper. Also we have made the mapping data available on an online repository which is planned to be updated on a regular basis. The results of our study seem to suggest that many software engineering techniques and activities are being used in the development of scientific software. However, there is still a need for further exploration of the usefulness of specific software engineering techniques (e.g., regarding software maintenance, evolution, refactoring, re(v)-engineering, process and project management) in the scientific context. It is hoped that this article will help (new) researchers get an overview of the research space and help them to understand the trends in the area.

A bibliometric analysis of the Turkish software engineering research community

“…Bibliometric rankings are quite common in field of SE, e.g., Caia and Card (2008), Wohlin (2007Wohlin ( , 2008, Glass and Chen (2001, 2003, Tse et al (2006), Wong et al (2008Wong et al ( , 2009Wong et al ( , 2011, Ren and Taylor (2007), Garousi and Varma (2010), , McCain et al (2005), Parnas (2007), Meyer et al (2009), Geist et al (1996, Farhoodi et al (2013), Garousi and Ruhe (2013), Jia and Harman (2010), Harman et al (2009). We discuss these works in the following.…”

“…We have also been able to find several systematic mapping and systematic literature review (SLR) studies, e.g., Farhoodi et al (2013), Jia and Harman (2010), Harman et al (2009) which, as part of their studies, have conducted a bibliometric assessment of papers in focused sub-areas of SE, namely: development of scientific software (Farhoodi et al 2013), software mutation testing (Jia and Harman 2010), and search-based SE (Harman et al 2009). These studies have measured and reported the number of publications by each country in these areas and provided interesting trends and insights.…”

“…The author of this study has recent experience in conducting systematic bibliometric studies, e.g., Farhoodi et al (2013), Garousi and Ruhe (2013), Garousi and Varma (2012). All that experience has enabled him to conduct this current bibliometric study with the same approach and rigor.…”

“…Similar to our previous bibliometric studies (Farhoodi et al 2013;Garousi and Ruhe 2013;Garousi and Varma 2012), the research approach that we use in our study is the goalquestion-metric (GQM) methodology (Basili 1992). Using the GQM's goal template (Basili 1992), the goal of this study is to study the volume of research contributions (i.e., number of papers) to the field of SE in the country of Turkey by various Turkish researchers and institutions for the purpose of ranking the output of researchers and institutions, from the point of view (and for the benefit) of young researchers, Ph.D. students and funding decision makers in Turkey.…”

“…There are various types of bibliometric studies in the field of SE addressing some of the above questions, e.g., Caia and Card (2008), Wohlin (2007Wohlin ( , 2008, Glass and Chen (2001, 2003, Tse et al (2006), Wong et al (2008Wong et al ( , 2009Wong et al ( , 2011, Ren and Taylor (2007), Garousi and Varma (2010), , McCain et al (2005), Parnas (2007), Meyer et al (2009), Geist et al (1996, Farhoodi et al (2013), Garousi and Ruhe (2013), Jia and Harman (2010), Harman et al (2009). For example, the on-going annual studies by Glass and Chen (2001, 2003, Tse et al (2006), Wong et al (2008Wong et al ( , 2009Wong et al ( , 2011 have been ranking the top researchers and institutions world-wide since 1996, addressing the questions (1) and (2) above.…”