Statistical errors in software engineering experiments

Reyes, Rolando P.; Dieste, Óscar; Fonseca, Efraín R.; Juristo, Natália

doi:10.1145/3180155.3180161

Cited by 10 publications

(6 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• We provide a more encouraging image of SE research than previous research [12,23]. ICSE is the flagship SE conference, but it has a general, i.e., non-experimental, character.…”

Section: Introductionmentioning

confidence: 88%

“…Construct validity: This threat operates when the study uses variables and metrics that do not represent the underlying theoretical constructs accurately. We have addressed this threat conducting previous research on (1) statistical errors in SE [23] and (2) experimental problems in the sciences (not published yet). These previous studies allowed us to design rigorous instruments (data collection forms, questionnaire); these instruments were also doublechecked and/or piloted.…”

Section: Threats To Validitymentioning

confidence: 99%

“…In a previous study [23], we searched for statistical errors in experiments published in the International Conference on Software Engineering (ICSE). We detected that a significant amount of experiments do not report statistical hypothesis, do not check the assumption of the inference tests, and perform multiple testing without any correction to avoid increasing type-I errors.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Publication Bias

Reyes¹,

Dieste²,

C.³

et al. 2020

Proceedings of the Evaluation and Assessment in Software Engineering

Self Cite

View full text Add to dashboard Cite

Background: Publication bias is the failure to publish the results of a study based on the direction or strength of the study findings. The existence of publication bias is firmly established in areas like medical research. Recent research suggests the existence of publication bias in Software Engineering. Aims: Finding out whether experiments published in the International Workshop on Empirical Software Engineering and Measurement (ESEM) are affected by publication bias. Method: We review experiments published in ESEM. We also survey with experimental researchers to triangulate our findings. Results: ESEM experiments do not define hypotheses and frequently perform multiple testing. One-tailed tests have a slightly higher rate of achieving statistically significant results. We could not find other practices associated with publication bias. Conclusions: Our results provide a more encouraging perspective of SE research than previous research: (1) ESEM publications do not seem to be strongly affected by biases and (2) we identify some practices that could be associated with p-hacking, but it is more likely that they are related to the conduction of exploratory research.

show abstract

“…• We provide a more encouraging image of SE research than previous research [12,23]. ICSE is the flagship SE conference, but it has a general, i.e., non-experimental, character.…”

Section: Introductionmentioning

confidence: 88%

Section: Threats To Validitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Publication Bias

Reyes¹,

Dieste²,

C.³

et al. 2020

Proceedings of the Evaluation and Assessment in Software Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…Because the purpose of the lookup table is to help Charlie choose a gas price for his transaction, we also assume that the estimated time of a transaction with a gas price 𝑥 𝑖 is always higher than the estimated time of another transaction with price 𝑥 𝑖+1 , for the same timestamp t. Finally we discretize prices into gas price categories to give a more practical, straightforward interpretation to the prices. We consider that [1,12] is very cheap, [13,24] is cheap, [25,36] is regular, [37,48] is expensive and [49,60] is very expensive. This allows Charlie to choose between prices to assign his transaction based on how quickly he wants it to be processed at that given time.…”

Section: Motivating Examplementioning

confidence: 99%

Is my transaction done yet? An empirical study of transaction processing times in the Ethereum Blockchain Platform

Pacheco¹,

Oliva²,

Rajbahadur³

et al. 2022

Preprint

View full text Add to dashboard Cite

Ethereum is one of the most popular platforms for the development of blockchain-powered applications. These applications are known as ÐApps. When engineering ÐApps, developers need to translate requests captured in the front-end of their application into one or more smart contract transactions. Developers need to pay for these transactions and, the more they pay (i.e., the higher the gas price), the faster the transaction is likely to be processed. Developing cost-effective ÐApps is far from trivial, as developers need to optimize the balance between cost (transaction fees) and user experience (transaction processing times). Online services have been developed to provide transaction issuers (e.g., ÐApp developers) with an estimate of how long transactions will take to be processed given a certain gas price. These estimation services are crucial in the Ethereum domain and several popular wallets such as Metamask rely on them. However, despite their key role, their accuracy has not been empirically investigated so far. In this paper, we quantify the transaction processing times in Ethereum, investigate the relationship between processing times and gas prices, and determine the accuracy of state-of-the-practice estimation services. Our results indicate that transactions are processed in a median of 57s and that 90% of the transactions are processed within 8m. We also show that higher gas prices result in faster transaction processing times with diminishing returns. In particular, we observe no practical difference in processing time between expensive and very expensive transactions. With regards to the accuracy of processing time estimation services, we observe that they are equivalent. However, when stratifying transactions by gas prices, we observe that Etherscan's Gas Tracker is the most accurate estimation service for very cheap and cheap transaction. EthGasStation's Gas Price API, in turn, is the most accurate estimation service for regular, expensive, and very expensive transactions. In a post-hoc study, we design a simple linear regression model with only one feature that outperforms the Gas Tracker for very cheap and cheap transactions and that performs as accurately as the EthGasStation model for the remaining categories. Based on our findings, ÐApp developers can make more informed decisions concerning the choice of the gas price of their application-issued transactions.

show abstract

“…A common mistake in data-driven software engineering is to fail to account for multiple hypothesis testing [19]. When simultaneously testing multiple hypotheses, some p-values can fall in the significance range by random chance.…”

Section: Multiplicity Of Hypothesis Testingmentioning

confidence: 99%

On the Impact of Programming Languages on Code Quality

Berger

Hollenbeck

Maj

et al. 2019

ACM Trans. Program. Lang. Syst.

View full text Add to dashboard Cite

This paper is a reproduction of work by Ray et al. which claimed to have uncovered a statistically significant association between eleven programming languages and software defects in projects hosted on GitHub. First we conduct an experimental repetition, repetition is only partially successful, but it does validate one of the key claims of the original work about the association of ten programming languages with defects. Next, we conduct a complete, independent reanalysis of the data and statistical modeling steps of the original study. We uncover a number of flaws that undermine the conclusions of the original study as only four languages are found to have a statistically significant association with defects, and even for those the effect size is exceedingly small. We conclude with some additional sources of bias that should be investigated in follow up work and a few best practice recommendations for similar efforts.ACM Reference Format:

show abstract

Statistical errors in software engineering experiments

Cited by 10 publications

References 80 publications

Publication Bias

Publication Bias

Is my transaction done yet? An empirical study of transaction processing times in the Ethereum Blockchain Platform

On the Impact of Programming Languages on Code Quality

Contact Info

Product

Resources

About