Refactorings are behavior-preserving program transformations that improve the design of a program. Refactoring engines are tools that automate the application of refactorings: first the user chooses a refactoring to apply, then the engine checks if the change is safe, and if so, transforms the program. Refactoring engines are a key component of modern IDEs, and programmers rely on them to perform refactorings. A fault in the refactoring engine can have severe consequences as it can erroneously change large bodies of source code and lead to strenuous debugging sessions.We present a technique for automated testing of refactoring engines. Test inputs for refactoring engines are programs. The core of our technique is a framework for iterative generation of structurally complex test inputs. We instantiate the framework to generate abstract syntax trees that represent Java programs. We also create several kinds of oracles to automatically check that the refactoring engine transformed the generated program correctly. We have applied our technique to testing Eclipse and NetBeans, two popular open-source IDEs for Java, and we have exposed new faults in both: 9 faults in Eclipse and 10 faults in NetBeans.

Abstract-Developers often change software in ways that cause tests to fail. When this occurs, developers must determine whether failures are caused by errors in the code under test or in the test code itself. In the latter case, developers must repair failing tests or remove them from the test suite. Reparing tests is time consuming but beneficial, since removing tests reduces a test suite's ability to detect regressions. Fortunately, simple program transformations can repair many failing tests automatically.We present ReAssert, a novel technique and tool that suggests repairs to failing tests' code which cause the tests to pass. Examples include replacing literal values in tests, changing assertion methods, or replacing one assertion with several. If the developer chooses to apply the repairs, ReAssert modifies the code automatically. Our experiments show that ReAssert can repair many common test failures and that its suggested repairs correspond to developers' expectations.

When developers change a program, regression tests can fail not only due to faults in the program but also due to outof-date test code that does not reflect the desired behavior of the program. When this occurs, it is necessary to repair test code such that the tests pass. Repairing tests manually is difficult and time consuming. We recently developed ReAssert, a tool that can automatically repair broken unit tests, but only if they lack complex control flow or operations on expected values.This paper introduces symbolic test repair, a technique based on symbolic execution, which can overcome some of ReAssert's limitations. We reproduce experiments from earlier work and find that symbolic test repair improves upon previously reported results both quantitatively and qualitatively. We also perform new experiments which confirm the benefits of symbolic test repair and also show surprising similarities in test failures for open-source Java and .NET programs. Our experiments use Pex, a powerful symbolic execution engine for .NET, and we find that Pex provides over half of the repairs possible from the theoretically ideal symbolic test repair.

Abstract. Bounded-exhaustive testing is an automated testing methodology that checks the code under test for all inputs within given bounds: first the user describes a set of test inputs and provides test oracles that check test outputs; then the tool generates all the inputs, executes them on the code under test, and checks the outputs; and finally the user inspects failing tests to submit bug reports. The costs of boundedexhaustive testing include machine time for test generation and execution (which translates into human time waiting for these results) and human time for inspection of results. This paper proposes three techniques that reduce these costs. Sparse Test Generation skips some tests to reduce the time to the first failing test. Structural Test Merging generates a smaller number of larger test inputs (rather than a larger number of smaller test inputs) to reduce test generation and execution time. Oracle-based Test Clustering groups failing tests to reduce the inspection time. Results obtained from the bounded-exhaustive testing of the Eclipse refactoring engine show that these three techniques can substantially reduce the costs while mostly preserving fault-detection capability.

To improve the overall user experience, graphical user interfaces (GUIs) of successful software systems evolve continuously. While the evolution is beneficial for end users, it creates several problems for developers and testers. Developers need to manually change the GUI code. Testers need to manually inspect and repair highly fragile test scripts. This is time-consuming and error-prone.The state-of-the-art tools for automatic GUI test repair use a black-box approach: they try to infer the changes between two GUI versions and then apply these changes to the test scripts. However, inferring these changes is challenging.We propose a white-box approach where the GUI changes are automated and knowledge about them is reused to repair the test cases appropriately. We use GUI refactorings as a means to encode the evolution of the GUIs. We envision a smart IDE that will record these refactorings precisely as they happen and will use them to change the GUI code and to repair test cases. We illustrate our approach through an example, discuss challenges that should be overcome to turn our vision into reality, and present a research agenda to address these challenges.