Sample aware embedded feature selection for reinforcement learning

Abstract

Reinforcement learning (RL) is designed to learn optimal control policies from unsupervised interactions with the environment. Many successful RL algorithms have been developed, however, none of them can efficiently tackle problems with high-dimensional state spaces due to the "curse of dimensionality", and so their applicability to real-world scenarios is limited. Here we propose a Sample Aware Feature Selection algorithm embedded in NEAT, or SAFS-NEAT, to help address this challenge. This algorithm builds upon the powerful evolutionary policy search algorithm NEAT, by exploiting data samples collected during the learning process. This data permits feature selection techniques from the supervised learning domain to be used to help RL scale to problems with high-dimensional state spaces. We show that by exploiting previously observed samples, on-line feature selection can enable NEAT to learn near optimal policies for such problems, and also outperform an existing feature selection algorithm which does not explicitly make use of this available data.

Predictive feature selection for genetic policy search

“…Increasing the number of lagged sensors (by altering the delay time) increases redundancy among features in the feature set, allowing the PFS-NEAT algorithm to easily detect and avoid adding many of them to the set. Loscalzo et al [33] showed that scaling the number of Gaussian random sensors provided very little impact on SAFS-NEAT, and our preliminary study revealed that more Gaussians do not present a challenge to PFS-NEAT either. As seen in the experimental section of that work, random sensors do have a strong negative impact on FS-NEAT.…”

“…The sample aware feature selection embedded in NEAT (SAFS-NEAT) algorithm represents the most closely related work to the proposed PFS-NEAT approach [33]. SAFS-NEAT attempts to overcome the high sample complexity of the IFSE-NEAT algorithm by altering how features are evaluated.…”

“…The majority of these works solely consider evolved policy performance to guide the feature selection process, and do not incorporate information from observed sample data that could be used to improve feature selection decisions [54][55][56]. Loscalzo et al [33] does drive the selection process using observed sample data, but the feature relevance measure used is domain-specific and is unlikely to yield acceptable results in general.…”

“…We empirically show that PFS-NEAT is able to learn near optimal control policies in both environments even as the number of irrelevant features increases. The algorithm is also able to identify feature subsets that are comprised of higher fractions of relevant sensors as compared to FS-NEAT [55], FD-NEAT [48], and SAFS-NEAT [33], which can cause PFS-NEAT to outperform these competitors. These experiments illustrate that PFS-NEAT is an effective feature selection algorithm that is suited to scaling up genetic policy search techniques to high-dimensional problems.…”

Optimization of Network Topology in Computer-Aided Detection Schemes Using Phased Searching with NEAT in a Time-Scaled Framework

“…In recent years, several variants of NEAT have been introduced and incorporated to feature selection or deselection (exclusion). 26,31,[53][54][55] For a more in-depth discussion on neuroevolution methods in the literature, the reader is referred to several literature reviews or surveys about the topic. 42,56 Phased searching with NeAt in a time-scaled Framework.…”