Unit Test Case Generation with Transformers and Focal Context

Abstract

Automated Unit Test Case generation has been the focus of extensive literature within the research community. Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult to read or understand for developers.In this paper we propose ATHENATEST, an approach that aims at generating unit test cases by learning from real-world, developer-written test cases. Our approach relies on a state-ofthe-art sequence-to-sequence transformer model which is able to write useful test cases for a given method under test (i.e., focal method). We also introduce METHODS2TEST-the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 630k test cases mined from 70k open-source repositories hosted on GitHub. We use this dataset to train a transformer model to translate focal methods into the corresponding test cases.We evaluate the ability of our model in generating test cases using natural language processing as well as code-specific criteria. First, we assess the quality of the translation compared to the target test case, then we analyze properties of the test case such as syntactic correctness and number and variety of testing APIs (e.g., asserts). We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3. Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated test cases.

TOGA: A Neural Method for Test Oracle Generation

“…In practice we found that both AthenaTests whole test generation and the seq2seq assertion model generated many tests and oracles that were not executable. The AthenaTest authors noted this issue in their evaluation, where they found that only 16% of the generated test cases were executable without errors and actively tested the unit under test [21]. The oracle generation model generated 34% executable oracles, and of these we observed that a further 5% were tautologies, resulting in an overall yield of 29% potentially meaningful oracles.…”

“…Lastly, AthenaTest [21] is a transformer model approach to generate entire unit tests including both prefixes and oracles. AthenaTest takes as input the unit's context (e.g., surrounding class, method signatures, etc.…”

“…Methods2Test* dataset. Our training dataset 𝐷 is variation of the Methods2Test dataset [21], we call Methods2Test*. As the name suggests, Methods2Test is a corpus of unit methods and corresponding developer written unit tests extracted from over 91K open source java projects.…”

“…To test neural methods, we compare with a seq2seq transformer (Codebert) [4] that is pretrained on 5 programming languages (including Java) and finetune it to generate assertions on the ATLAS assertion dataset. We also evaluate against a whole-test generation model, AthenaTest [21]. (3) Specification Mining.…”

“…Recently, neural generative models have shown promise in generating functional test oracles [21][22][23]. Neural methods are more flexible than specification mining approaches as they do not rely on fixed patterns.…”