Empirical Strategies in Software Engineering Research: A Literature Survey

Guevara‐Vega, Cathy; Bernárdez, Beatriz; Durán, Amador; Quiña‐Mera, Antonio; Cruz, Margarita; Ruiz–Cortés, Antonio

doi:10.1109/ici2st51859.2021.00025

Cited by 9 publications

(11 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent survey closely related to the approach presented in this article, 137 articles published between 2013 and 2018 reporting at least one replication were analyzed [18]. Most of the replication changes were defined in controlled experiments, confirming that this is the most frequently applied empirical strategy in the research area [29]. In general, most changes-also referred to as adjustments or differences-were described with respect to an original experiment using natural language or tabular forms, only about 25-30% of the studies reported the purpose of the changes and their effect on experimental validity.…”

Section: Related Workmentioning

confidence: 54%

A model-based approach for specifying changes in replications of empirical studies in computer Science

Cruz¹,

Bernárdez²,

Durán³

et al. 2022

Computing

Self Cite

View full text Add to dashboard Cite

The need of replicating empirical studies in Computer Science is widely recognized among the research community. It is essential to report the changes of each replication to promote not only the comprehensibility of the evolution of the experimental validity across a family of studies, but also replicability itself. Unfortunately, the lack of proposals for systematic reporting of changes in replications undermines these desirable objectives. The main goal of the work presented in this article is to provide researchers in Computer Science with a systematic tool-supported approach for the specification and reporting of changes in the replications of their empirical studies. Applying Design Science Research, we have developed and validated a composite artifact consisting of (i) a metamodel that formalizes all the relevant concepts related to replications and their changes; (ii) templates and linguistic patterns that facilitate their reporting; and (iii) a proof-of-concept model-based software tool that supports the proposed approach. For its validation, we have carried out a multiple case study that includes 9 families of empirical studies not only from Computer Science, but also from an area as different as Agrobiology , to check the external validity of our approach. The 9 families encompass 23 replication studies and a total of 92 replication changes, for which we have analyzed the suitability of our proposal. The multiple case study revealed some initial limitations of our approach related to threats to experimental validity and context variables. After several improvement iterations on the artifact, all of the 92 replication changes could be properly specified, including also their qualitatively estimated effects on experimental validity and their corresponding visualization. Our proposal for the specification of replication changes seems to fit the needs not only of replications in Computer Science, but also in other research areas. Nevertheless, further research is needed to improve it and disseminate its use among the research community.

show abstract

Section: Related Workmentioning

confidence: 54%

A model-based approach for specifying changes in replications of empirical studies in computer Science

Cruz¹,

Bernárdez²,

Durán³

et al. 2022

Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…We found five publications on empirical research in RE published between 2005 and 2016 and nine on empirical research in SE published between 2002 and 2021. While one publication [1] examined empirical research in RE using a survey with 42 respondents, the other 13 publications [2], [3], [5]- [15] used (systematic) literature reviews or systematic mapping studies to analyze on average 402.9 papers (minimum: 20, median: 154, and maximum: 2237 papers) published between 1977 and 2019 with overlapping periods. In total, these 13 publications examined papers from a total of 60 different venues on 18 different themes.…”

Section: Related Workmentioning

confidence: 99%

“…This overlap and redundancy could have been avoided if researchers had collaborated to build on and update earlier works. However, only four out of 14 publications offer their data at all, with only one publication [15] using a public data repository [71], [72]. The other three publications only offer links that no longer work [5], [12], [13].…”

Section: Related Workmentioning

confidence: 99%

Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering

Karras,

Wernlein,

Klünder

et al. 2023

2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

View full text Add to dashboard Cite

Background.] Empirical research in requirements engineering (RE) is a constantly evolving topic, with a growing number of publications. Several papers address this topic using literature reviews to provide a snapshot of its "current" state and evolution. However, these papers have never built on or updated earlier ones, resulting in overlap and redundancy. The underlying problem is the unavailability of data from earlier works. Researchers need technical infrastructures to conduct sustainable literature reviews. [Aims.] We examine the use of the Open Research Knowledge Graph (ORKG) as such an infrastructure to build and publish an initial Knowledge Graph of Empirical research in RE (KG-EmpiRE) whose data is openly available. Our long-term goal is to continuously maintain KG-EmpiRE with the research community to synthesize a comprehensive, up-todate, and long-term available overview of the state and evolution of empirical research in RE. [Method.] We conduct a literature review using the ORKG to build and publish KG-EmpiRE which we evaluate against competency questions derived from a published vision of empirical research in software (requirements) engineering for 2020 -2025. [Results.] From 570 papers of the IEEE International Requirements Engineering Conference (2000 -2022), we extract and analyze data on the reported empirical research and answer 16 out of 77 competency questions. These answers show a positive development towards the vision, but also the need for future improvements. [Conclusions.] The ORKG is a ready-to-use and advanced infrastructure to organize data from literature reviews as knowledge graphs. The resulting knowledge graphs make the data openly available and maintainable by research communities, enabling sustainable literature reviews.

show abstract

“…Moreover, participants are often biased in their responses, which is both a desirable treat (since it allows assessing human perception), but also a threat to validity (since it can affect the responses) [35]. Although open questions allow more natural responses, even a strict coding analysis will be subjected to researcher bias [38].…”

Section: Survey Constructionmentioning

confidence: 99%

On technical debt in mathematical programming: An exploratory study

Vidoni

Cúnico

2022

Math. Prog. Comp.

View full text Add to dashboard Cite

The Technical Debt (TD) metaphor describes development shortcuts taken for expediency that cause the degradation of internal software quality. It has served the discourse between engineers and management regarding how to invest resources in maintenance and extend into scientific software (both the tools, the algorithms and the analysis conducted with it). Mathematical programming has been considered ‘special purpose programming’, meant to program and simulate particular problem types (e.g., symbolic mathematics through Matlab). Likewise, more traditional mathematical programming has been considered ‘modelling programming’ to program models by providing programming structures required for mathematical formulations (e.g., GAMS, AMPL, AIMMS). Because of this, other authors have argued the need to consider mathematical programming as closely related to software development. As a result, this paper presents a novel exploration of TD in mathematical programming by assessing self-reported practices through a survey, which gathered 168 complete responses. This study discovered potential debts manifested through smells and attitudinal causes towards them. Results uncovered a trend to refactor and polish the final mathematical model and use version control and detailed comments. Nonetheless, we uncovered traces of negative practices regarding Code Debt and Documentation Debt, alongside hints indicating that most TD is deliberately introduced (i.e., modellers are aware that their practices are not the best). We aim to discuss the idea that TD is also present in mathematical programming and that it may hamper the reproducibility and maintainability of the models created. The overall goal is to outline future areas of work that can lead to changing current modellers’ habits and assist in extending existing mathematical programming (both practice and research) to eventually manage TD in mathematical programming.

show abstract

Empirical Strategies in Software Engineering Research: A Literature Survey

Cited by 9 publications

References 47 publications

A model-based approach for specifying changes in replications of empirical studies in computer Science

A model-based approach for specifying changes in replications of empirical studies in computer Science

Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering

On technical debt in mathematical programming: An exploratory study

Contact Info

Product

Resources

About