Empirical evaluation of smart contract testing: what is the best choice?

Ren, Meng; Yin, Zijing; Ma, Fuchen; Xu, Zhenyang; Jiang, Yu; Sun, Chengnian; Li, Huizhong; Cai, Yan

doi:10.1145/3460319.3464837

Cited by 59 publications

(32 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, other researchers performed empirical studies as shown in Table 2 to evaluate and compare the smart contract vulnerability detectors. 2020 [11] 2020 [16] 2021 [10]…”

Section: Empirical Evaluations Of Vulnerability Detectorsmentioning

confidence: 99%

“…Many methods and corresponding tools, e.g., Oyente [2], Securify [3], sFuzz [4], ContractFuzzer [5], ContraMaster [6], DefectChecker [7], EXGEN [8], HONEYBADGER [9], have been proposed to detect smart contract vulnerabilities. Ren et al [10] point out that the tools are evaluated by the tool authors using different datasets and metrics, which may result in biased conclusions. Several empirical studies were conducted by other researchers using manually annotated datasets or real-world smart contracts to evaluate these tools fairly.…”

Section: Introductionmentioning

confidence: 99%

“…• Last but not least, we have created a benchmark dataset containing 4,364 real-world Solidity smart contracts, which are manually labeled with ten types of vulnerabilities. The dataset is around 20 times bigger than the similar stateof-the-art benchmark [10]. The dataset can help evaluate the vulnerability detector tools more consistently and objectively and is available at https://github.com/1052445594/SC UEE.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Why Smart Contracts Reported as Vulnerable were not Exploited?

Hu¹,

Li²,

Storhaug³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>As smart contracts process digital assets, their security is essential for blockchain applications. Many approaches have been proposed to detect smart contract vulnerabilities. Studies show that few of the reported vulnerabilities are exploited and hypothesize that many of the reported vulnerabilities are false positives. However, no follow-up study is performed to confirm the hypothesis and understand why the reported vulnerabilities are not exploited. In this study, we first collect 136,969 unique real-world smart contracts and analyze them using four vulnerability detectors, namely Oyente, SmartCheck, Slither, and SolDetector. Then, we apply Strauss’ grounded theory approach to manually analyze the source code of the smart contracts reported as vulnerable to recognizing false positives and understand the reasons for false results. In addition, we analyze the transaction logs of the smart contracts reported as vulnerable to identifying and understanding their exploitations. Our results show that 75.37% of the 4,364 smart contracts reported as vulnerable are false positives, and eleven reasons are causing the false positives. After analyzing the 4,106,134 transaction logs of the contracts reported as vulnerable, we find that vulnerabilities of only 67 (0.015%) of the contracts have been exploited in history. We also identify six reasons for demotivating and preventing the attackers from exploiting the vulnerabilities. Our results reveal that state-of-the-art smart contract vulnerability detectors primarily treat the smart contracts as yet another application developed using Object Oriented (OO) languages when analyzing and reporting the smart contract vulnerabilities. Without considering the specific design principles of the Solidity programming language and the characteristics of smart contracts' application scenarios and execution environments, many of the reported vulnerabilities are not exploitable or not cost-effective to be exploited by adversaries. </p>

show abstract

“…Thus, other researchers performed empirical studies as shown in Table 2 to evaluate and compare the smart contract vulnerability detectors. 2020 [11] 2020 [16] 2021 [10]…”

Section: Empirical Evaluations Of Vulnerability Detectorsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Why Smart Contracts Reported as Vulnerable were not Exploited?

Hu¹,

Li²,

Storhaug³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This includes symbolic execution tools, like Maian [28], Manticore [26], Oyente [22], Zeus [18], Securify [36], Mythril [27]. Some tools rely on exhaustive state exploration, via model checking or SMT solving, sometimes leading to a slow analysis [1], timeouts [31], or lack of results [1]. [18] relies on Abstract Interpretation for a preliminary analysis, and uses an SMT solver to check properties on inferred invariants.…”

Section: Related Workmentioning

confidence: 99%

Abstract interpretation of Michelson smart-contracts

Bau¹,

Miné

Botbol³

et al. 2022

Proceedings of the 11th ACM SIGPLAN International Workshop on the State of the Art in Program Analysis

View full text Add to dashboard Cite

“…In the papers [17], [19], [29], [30], [31], [34], [45], and [61], a newer method called mutation testing was introduced for the specific purpose of testing smart contracts. In addition, in the papers [20], [21], [23], [24], [26], [41], [42], [48], [53], [54], [55], and [56], another method called fuzz testing was introduced as well.…”

Section: ) Testing Data Challengesmentioning

confidence: 99%

Systematic Mapping of Testing Smart Contracts for Blockchain Applications

Imperius

Alahmar

2022

IEEE Access

View full text Add to dashboard Cite

In the last few years, the technological future becoming apparent by the introduction of smart contracts into mainstream technology, specifically in the development of Web3 and the metaverse. Smart contracts will play a vital role in the decentralization and autonomy of the day-to-day tasks that must be completed. Several literature reviews, considered secondary sources, highlight the current state of testing methods for smart contracts made for Blockchain applications. In this paper, we present the results from a systematic mapping study to give structure to the information found from primary sources. Systematic mapping is a well-known method to identify and categorize research papers in a field with an increasing amount of literature. For this systematic mapping, we searched for studies between 2017 and present-day (March 2022) and were able to find 303 results, from which 47 were selected, by specific inclusion and exclusion criteria, to be relevant to this study. A concept map was created from the information gathered from primary sources to the attributes such as research type, contribution type, blockchain network, smart contract language, development process, testing methods, and testing environment. We also categorized the trends and demographics found in the selected papers based on publication year, author's country, and more. The results of this systematic mapping showed that this field is very new and quickly increasing with new research. The researchers that are interested in this field could use the results found to create opportunities for their future work.

show abstract

Empirical evaluation of smart contract testing: what is the best choice?

Cited by 59 publications

References 40 publications

Why Smart Contracts Reported as Vulnerable were not Exploited?

Why Smart Contracts Reported as Vulnerable were not Exploited?

Abstract interpretation of Michelson smart-contracts

Systematic Mapping of Testing Smart Contracts for Blockchain Applications

Contact Info

Product

Resources

About