Improving generalization of evolved programs through automatic simplification

Helmuth, Thomas; McPhee, Nicholas Freitag; Pantridge, Edward; Spector, Lee

doi:10.1145/3071178.3071330

Cited by 53 publications

(28 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results are particular to Push program solutions, but should correlate with the sizes of programs needed to solve these problems in other systems. In order to find each size, we took each solution program and automatically simplified it to produce a smaller equivalent program [14]. Of these simplified programs, in Table 3 we report the smallest simplified solution size out of all simplified solutions to each problem.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Psb2

Helmuth

Kelly

2021

Proceedings of the Genetic and Evolutionary Computation Conference

Self Cite

View full text Add to dashboard Cite

For the past six years, researchers in genetic programming and other program synthesis disciplines have used the General Program Synthesis Benchmark Suite to benchmark many aspects of automatic program synthesis systems. These problems have been used to make notable progress toward the goal of general program synthesis: automatically creating the types of software that human programmers code. Many of the systems that have attempted the problems in the original benchmark suite have used it to demonstrate performance improvements granted through new techniques. Over time, the suite has gradually become outdated, hindering the accurate measurement of further improvements. The field needs a new set of more difficult benchmark problems to move beyond what was previously possible.In this paper, we describe the 25 new general program synthesis benchmark problems that make up PSB2, a new benchmark suite. These problems are curated from a variety of sources, including programming katas and college courses. We selected these problems to be more difficult than those in the original suite, and give results using PushGP showing this increase in difficulty. These new problems give plenty of room for improvement, pointing the way for the next six or more years of general program synthesis research. CCS CONCEPTS• Software and its engineering → Automatic programming.

show abstract

Section: Resultsmentioning

confidence: 99%

“…As has been shown to be effective at improving generalization, we use an automatic simplification procedure on every evolved Push program that passes all of the training cases before testing it on the test set [14].…”

Section: Experimental Methods and System Parametersmentioning

confidence: 99%

Psb2

Helmuth

Kelly

2021

Proceedings of the Genetic and Evolutionary Computation Conference

Self Cite

View full text Add to dashboard Cite

show abstract

“…Given a list of elements that are all of the same comparable type, T , and two instance of type T representing a lower and upper bound, filter the list to the elements that fall between two bounds (inclusively). For example, if given the list [6,5,4,3,2,1], the lower bound 3, and the upper bound 5 the result should be [5,4,3]. Also, given the list ["a","b","c"], the lower bound "x", and the upper bound "zzz" the result should be an empty list.…”

Section: Filter Boundsmentioning

confidence: 99%

Code building genetic programming

Pantridge

Spector

2020

Proceedings of the 2020 Genetic and Evolutionary Computation Conference

Self Cite

View full text Add to dashboard Cite

In recent years the field of genetic programming has made significant advances towards automatic programming. Research and development of contemporary program synthesis methods, such as PushGP and Grammar Guided Genetic Programming, can produce programs that solve problems typically assigned in introductory academic settings. These problems focus on a narrow, predetermined set of simple data structures, basic control flow patterns, and primitive, non-overlapping data types (without, for example, inheritance or composite types). Few, if any, genetic programming methods for program synthesis have convincingly demonstrated the capability of synthesizing programs that use arbitrary data types, data structures, and specifications that are drawn from existing codebases. In this paper, we introduce Code Building Genetic Programming (CBGP) as a framework within which this can be done, by leveraging programming language features such as reflection and first-class specifications. CBGP produces a computational graph that can be executed or translated into source code of a host language. To demonstrate the novel capabilities of CBGP, we present results on new benchmarks that use non-primitive, polymorphic data types as well as some standard program synthesis benchmarks.

show abstract

“…Additionally, for a program to count as a generalizing success, it must perfectly pass a large number of withheld test 1 https://github.com/lspector/Clojush 2 https://github.com/thelmuth/Clojush/releases/tag/Novelty-Lexicase cases not used during evolution. Before testing for generalization, we automatically simplify each solution program, which has been shown to significantly raise generalization rates [7]. To test for significant differences in success rates, we use a pairwise chi-square test with Holm correction and a 0.05 significance level.…”

Section: Gp System and Parametersmentioning

confidence: 99%

Comparing and combining lexicase selection and novelty search

Jundt

Helmuth

2019

Proceedings of the Genetic and Evolutionary Computation Conference

Self Cite

View full text Add to dashboard Cite

Lexicase selection and novelty search, two parent selection methods used in evolutionary computation, emphasize exploring widely in the search space more than traditional methods such as tournament selection. However, lexicase selection is not explicitly driven to select for novelty in the population, and novelty search suffers from lack of direction toward a goal, especially in unconstrained, highly-dimensional spaces. We combine the strengths of lexicase selection and novelty search by creating a novelty score for each test case, and adding those novelty scores to the normal error values used in lexicase selection. We use this new novelty-lexicase selection to solve automatic program synthesis problems, and find it significantly outperforms both novelty search and lexicase selection. Additionally, we find that novelty search has very little success in the problem domain of program synthesis. We explore the effects of each of these methods on population diversity and long-term problem solving performance, and give evidence to support the hypothesis that novelty-lexicase selection resists converging to local optima better than lexicase selection.

show abstract

Improving generalization of evolved programs through automatic simplification

Cited by 53 publications

References 25 publications

Psb2

Psb2

Code building genetic programming

Comparing and combining lexicase selection and novelty search

Contact Info

Product

Resources

About