Characterizing the Effects of Random Subsampling on Lexicase Selection

Ferguson, Austin J.; Hernandez, Jose Guadalupe; Junghans, Daniel; Lalejini, Alexander; Dolson, Emily; Ofria, Charles

doi:10.1007/978-3-030-39958-0_1

Cited by 23 publications

(47 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For these reasons, lexicase selection can be thought of as more faithfully modeling interactions between biological organisms and their environments. Hernandez et al (2019) recently proposed two methods for subsampling the training set each generation when using lexicase selection, which were further studied by Ferguson et al (2019). Down-sampled lexicase selection uses a different random subsample of cases for each generation.…”

Section: Introductionmentioning

confidence: 99%

“…These computational savings can be recouped by evaluating more individuals throughout evolution. Results from Hernandez et al (2019) and Ferguson et al (2019) indicate that both of these methods improve problem-solving performance compared to standard lexicase selection.…”

Section: Introductionmentioning

confidence: 99%

“…In this article we concentrate on down-sampled lexicase selection, as it is simpler in concept and implementation, and both Hernandez et al (2019) and Ferguson et al (2019) found its benefits to be comparable to cohort lexicase selection. We first conduct a more expansive benchmarking of downsampled lexicase selection than has been conducted previously, using more benchmark problems and subsample sizes.…”

Section: Introductionmentioning

confidence: 99%

“…We then turn to developing a better understanding of why down-sampled lexicase selection performs so well. One hypothesis put forward by Ferguson et al (2019) is that down-sampled lexicase selectionʼs success hinges on it enabling deeper evolutionary searches for more generations given the same computational effort. We compare this hypothesis to the hypothesis that simply evaluating more individuals in the search space is more important than deeper evolution specifically.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Problem-Solving Benefits of Down-Sampled Lexicase Selection

Helmuth

Spector

2021

Artificial Life

View full text Add to dashboard Cite

In genetic programming, an evolutionary method for producing computer programs that solve specified computational problems, parent selection is ordinarily based on aggregate measures of performance across an entire training set. Lexicase selection, by contrast, selects on the basis of performance on random sequences of training cases; this has been shown to enhance problem-solving power in many circumstances. Lexicase selection can also be seen as better reflecting biological evolution, by modeling sequences of challenges that organisms face over their lifetimes. Recent work has demonstrated that the advantages of lexicase selection can be amplified by down-sampling, meaning that only a random subsample of the training cases is used each generation. This can be seen as modeling the fact that individual organisms encounter only subsets of the possible environments and that environments change over time. Here we provide the most extensive benchmarking of down-sampled lexicase selection to date, showing that its benefits hold up to increased scrutiny. The reasons that down-sampling helps, however, are not yet fully understood. Hypotheses include that down-sampling allows for more generations to be processed with the same budget of program evaluations; that the variation of training data across generations acts as a changing environment, encouraging adaptation; or that it reduces overfitting, leading to more general solutions. We systematically evaluate these hypotheses, finding evidence against all three, and instead draw the conclusion that down-sampled lexicase selection's main benefit stems from the fact that it allows the evolutionary process to examine more individuals within the same computational budget, even though each individual is examined less completely.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Problem-Solving Benefits of Down-Sampled Lexicase Selection

Helmuth

Spector

2021

Artificial Life

View full text Add to dashboard Cite

show abstract

“…For reasons discussed in Section 3, we have created datasets consisting of large numbers of inputs and correct outputs for every problem [13]. 5 The dataset for each problem consists of a small number of hand-chosen inputs, often addressing edge cases for the problem, and 1 million randomly-generated inputs falling within the constraints of the problem. We recommend each different program synthesis run use a different set of data, composed of every one of the hand-chosen inputs and a random sample of the randomly-generated inputs.…”

Section: Using Psb2mentioning

confidence: 99%

Psb2

Helmuth

Kelly

2021

Proceedings of the Genetic and Evolutionary Computation Conference

View full text Add to dashboard Cite

For the past six years, researchers in genetic programming and other program synthesis disciplines have used the General Program Synthesis Benchmark Suite to benchmark many aspects of automatic program synthesis systems. These problems have been used to make notable progress toward the goal of general program synthesis: automatically creating the types of software that human programmers code. Many of the systems that have attempted the problems in the original benchmark suite have used it to demonstrate performance improvements granted through new techniques. Over time, the suite has gradually become outdated, hindering the accurate measurement of further improvements. The field needs a new set of more difficult benchmark problems to move beyond what was previously possible.In this paper, we describe the 25 new general program synthesis benchmark problems that make up PSB2, a new benchmark suite. These problems are curated from a variety of sources, including programming katas and college courses. We selected these problems to be more difficult than those in the original suite, and give results using PushGP showing this increase in difficulty. These new problems give plenty of room for improvement, pointing the way for the next six or more years of general program synthesis research. CCS CONCEPTS• Software and its engineering → Automatic programming.

show abstract