On the analysis and design of software for reinforcement learning, with a survey of existing systems

Abstract

Reinforcement Learning (RL) is a very complex domain and software for RL is correspondingly complex. We analyse the scope, requirements, and potential for RL software, discuss relevant design issues, survey existing software, and make recommendations for designers. We argue that broad and flexible libraries of reusable software components are valuable from a scientific, as well as practical, perspective, as they allow precise control over experimental conditions, encourage comparison of alternative methods, and allow a fuller exploration of the RL domain.

RLLib: C++ Library to Predict, Control, and Represent Learnable Knowledge Using On/Off Policy Reinforcement Learning

“…RL toolbox 5 , libpgrl 6 , YORLL 7 , and rllib 8 [8] are C++ based platforms to develop RL algorithms in different scenarios, while CLSquare 9 [9] is a standardized platform for testing RL problems with on-policy batch controllers. BURLAP 10 [7], PIQLE 11 [6], MMF 12 , QCON 13 , and RLPark 14 are Java platforms that model and learn from RL problems. MDP Toolbox 15 is an Octave based RL development platform.…”

“…RLLib closely follows the design principles and recommendations presented in [13,24]. The development of the library has taken significant efforts to minimize memory footprint as well as computational requirements that are requested by RL problems.…”

Relational approach to knowledge engineering for POMDP-based assistance systems as a translation of a psychological model

“…We start this section with a reference to a more general approach of reinforcement learning (RL) which represents a broader class of planning problems where available domain knowledge is sufficient for only a partial formulation of the problem and the planning algorithm has to estimate the missing elements using simulation (Bertsekas and Tsitsiklis, 1996). A recent survey of the existing tools and software for engineering RL problem specifications presented in (Kovacs and Egginton, 2011) shows that engineering of RL domains is in most cases done either by re-implementation of required algorithms and domains/simulators or by partial re-use of the existing source code in the form of libraries or repositories, which means that engineering of RL domains is a direct implementation problem. There are no existing out of the box, domain independent environments where the specification of the RL problem would be reduced to the specification of the domain in a specific domain definition language.…”