<i>ConTesa</i>: Directed Test Suite Augmentation for Concurrent Software

Yu, Tingting; Huang, Zunchen; Wang, Chao

doi:10.1109/tse.2018.2861392

Cited by 7 publications

(3 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use mutmut [9], a tool having over 600 stars on GitHub, to conduct mutation testing for Python solutions. Mutmut is widely adopted for mutation testing research on Python [15,22,24,71]. We use Mull [17,46] to conduct mutation testing for C++ solutions, which is also a widely used tool with more than 600 stars on GitHub.…”

Section: Mutation Score Analysismentioning

confidence: 99%

Who Judges the Judge: An Empirical Study on Online Judge Tests

Liu

Yan

Zhang

et al. 2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Online Judge platforms play a pivotal role in education, competitive programming, recruitment, career training, and large language model training. They rely on predefined test suites to judge the correctness of submitted solutions. It is therefore important that the solution judgement is reliable and free from potentially misleading false positives (i.e., incorrect solutions that are judged as correct). In this paper, we conduct an empirical study of 939 coding problems with 541,552 solutions, all of which are judged to be correct according to the test suites used by the platform, finding that 43.1% of the problems include false positive solutions (3,700 bugs are revealed in total). We also find that test suites are, nevertheless, of high quality according to widely-studied test effectiveness measurements: 88.2% of false positives have perfect (100%) line coverage, 78.9% have perfect branch coverage, and 32.5% have a perfect mutation score. Our findings indicate that more work is required to weed out false positive solutions and to further improve test suite effectiveness. We have released the detected false positive solutions and the generated test inputs to facilitate future research. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

show abstract

Section: Mutation Score Analysismentioning

confidence: 99%

Who Judges the Judge: An Empirical Study on Online Judge Tests

Liu

Yan

Zhang

et al. 2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

show abstract

“…The second category of dynamic analyses focuses on how to trigger concurrency violation conditions. This includes random testings that mimic non-deterministic program executions [4,25,38], regression testings [47,60] that target interleavings from code changes, model checking [13,57,62] and hybrid constraint solving [20][21][22] approaches that systematically check or execute possible thread schedules, heuristically avoid fruitless executions [10,17,18,66], or utilizing multicore to accelerate bug detection [37]. Our work differs from all the above, as our focus is not to test schedules with a given seed file, but to generate seed files that execute multithreadingrelevant paths.…”

Section: Dynamic Analysis On Concurrency-bugsmentioning

confidence: 99%

MUZZ: Thread-aware Grey-box Fuzzing for Effective Bug Hunting in Multithreaded Programs

Chen,

Guo,

Xue

et al. 2020

Preprint

View full text Add to dashboard Cite

Grey-box fuzz testing has revealed thousands of vulnerabilities in real-world software owing to its lightweight instrumentation, fast coverage feedback, and dynamic adjusting strategies. However, directly applying grey-box fuzzing to input-dependent multithreaded programs can be extremely inefficient. In practice, multithreading-relevant bugs are usually buried in the sophisticated program flows. Meanwhile, existing grey-box fuzzing techniques do not stress threadinterleavings that affect execution states in multithreaded programs. Therefore, mainstream grey-box fuzzers cannot adequately test problematic segments in multithreaded software, although they might obtain high code coverage statistics.To this end, we propose MUZZ, a new grey-box fuzzing technique that hunts for bugs in multithreaded programs. MUZZ owns three novel thread-aware instrumentations, namely coverage-oriented instrumentation, thread-context instrumentation, and schedule-intervention instrumentation. During fuzzing, these instrumentations engender runtime feedback to accentuate execution states caused by thread interleavings. By leveraging such feedback in the dynamic seed selection and execution strategies, MUZZ preserves more valuable seeds that expose bugs under a multithreading context.We evaluate MUZZ on twelve real-world multithreaded programs. Experiments show that MUZZ outperforms AFL in both multithreading-relevant seed generation and concurrency-vulnerability detection. Further, by replaying the target programs against the generated seeds, MUZZ also reveals more concurrency-bugs (e.g., data-races, thread-leaks) than AFL. In total, MUZZ detected eight new concurrencyvulnerabilities and nineteen new concurrency-bugs. At the time of writing, four reported issues have received CVE IDs.

show abstract

“…The other dynamic analyses focus on how to trigger the concurrency violation conditions. These include the random testings that mimic non-deterministic program executions [114][115][116][117][118], the regression testing methods [119,120] that targets interleavings from code changes, the model checking [121][122][123][124] and hybrid constraint solving [104,125,126] approaches that systematically checks or execute possible thread schedules, or heuristically avoid fruitless executions [127,128]. Our work differs from all of them in the sense that our goal is to cover more execution paths by generating different seeds that exercise different paths in multithreading-relevant settings.…”

Section: Dynamic Analysis On Concurrency Bugsmentioning

confidence: 99%

Securing software systems via fuzz testing and verification

Chen¹

View full text Add to dashboard Cite

Software security has been growing in importance due to the increasing reliance on various systems such as computers, smartphones, and Internet-of-Things devices.Security weaknesseses, or vulnerabilities, range in various aspects, from the systematic levels such as designing defects, to implementation errors such as language-specific program crashes. The multifaceted nature of vulnerabilities poses significant challenges to secure different systems. In fact, multilevel approaches have to be applied to reveal vulnerabilities or solidify the systems. Generally, there are two categories of approaches to securing a system, namely testing and verification. The former tries to generate test cases to trigger erroneous behaviors in the underlying programs. The latter models the system in an abstracted description to prove the absence of certain security defects. In this thesis, we will describe our efforts in greybox fuzz testing and type checking based verification in securing different levels of software systems.Firstly, we apply greybox fuzz testing (or fuzzing) to detect implementation vulnerabilities. These vulnerabilities are usually subtle that it is rather complicated to formally verify the absence of the corresponding defects. We target on two levels of vulnerabilities. One is to improve the fuzzing directedness of executing towards specific target program locations to reveal crash induced vulnerabilities such as buffer-overflow, use-after-free, etc. To emphasize existing challenges in directed fuzzing, we propose HAWKEYE to feature four desired properties for directedness. With the statically analyzed results on the target program and its target locations, HAWKEYE applies multiple dynamic strategies for seed prioritization, power scheduling and mutation. The experimental results on various real-world programs showed that HAWKEYE significantly outperforms the state-of-the-art greybox fuzzers AFL and AFLGo in reaching i Abstract ii target locations and reproducing specific crashes. The other scenario is to enhance fuzzing to reveal multithreading-relevant bugs. These bugs may root from logic errors which exhibit abnormal behaviors in the target program. They may or may not cause immediate crashes however implicate certain security defects. We present DOU-BLADE, a novel thread-aware technique which effectively generates multithreadingrelevant seeds. DOUBLADE relies on a set of thread-aware techniques, consisting of a stratified exploration-oriented instrumentation and two complementary instrumentations, as well as dynamic adaptive strategies based on feedbacks. Experiments on 12

show abstract

ConTesa: Directed Test Suite Augmentation for Concurrent Software

Cited by 7 publications

References 57 publications

Who Judges the Judge: An Empirical Study on Online Judge Tests

Who Judges the Judge: An Empirical Study on Online Judge Tests

MUZZ: Thread-aware Grey-box Fuzzing for Effective Bug Hunting in Multithreaded Programs

Securing software systems via fuzz testing and verification

Contact Info

Product

Resources

About