MoËT: Mixture of Expert Trees and its application to verifiable reinforcement learning

Vasić, Marko; Petrović, Andrija; Wang, Kaiyuan; Nikolić, Mladen; Singh, Rishabh; Khurshid, Sarfraz

doi:10.1016/j.neunet.2022.03.022

Cited by 19 publications

(7 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During the learning process, this clustering became more distinct, and in the 'expert' agent the clusters were associated with specific phases of θ during the circling motion around the thermal center. Further analyzing these clusters and their evolution during the learning process may inform us about the inner workings of the NN and may be used to understand the agent's policy by extracting explicit rules, for example, by describing each cluster as a decision tree [39][40][41][42] . In summary, the application of deep-RL method as presented here may contribute to further improving autonomous UAV gliding systems and to our understanding of motion control learning.…”

Section: Discussionmentioning

confidence: 99%

Revealing principles of autonomous thermal soaring in windy conditions using vulture-inspired deep reinforcement-learning

Flato,

Harel,

Tamar

et al. 2023

Preprint

View full text Add to dashboard Cite

Thermal soaring, a technique used by birds and gliders to utilize updrafts of hot air, presents an attractive model for developing biomimetic autonomous and unmanned aerial vehicles (UAVs) capable of long-endurance flight. Previous studies have employed machine- and deep-learning models to control gliding UAVs in simplified environments without horizontal winds. The resulting neural network models operate as 'black boxes', with little insight into their navigation policy or the structure of the problem. Here, we present a deep reinforcement-learning framework for autonomous glider control in a simulated environment with thermal updrafts and challenging horizontal winds. Compared with vulture flight data, the resulting autonomous agent spontaneously adopted a vulture-like soaring technique that successfully and robustly exploited thermal updrafts under horizontal winds up to 5 m/sec. This system enabled us to reveal the underlying structure of the thermal soaring problem, which consists of two critical bottlenecks that should be solved sequentially: achieving stable flight and flying near the thermal center. Additionally, the agent's neural network divides into functional clusters that correlate with distinct behavioral modes during thermal searching and soaring. Our findings may contribute to the development of biomimetic UAVs with vulture-like efficiency and to understanding the structure and bottlenecks of other motion-based problems.

show abstract

Section: Discussionmentioning

confidence: 99%

Revealing principles of autonomous thermal soaring in windy conditions using vulture-inspired deep reinforcement-learning

Flato,

Harel,

Tamar

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The mechanistic interpretability of a standard deep-learning approach (cf. [281][282][283][284][285][286][287][288][289][290][291][292]) is limited with nearly a black box and one typically not amenable to the individual differences here given model demands for data and dimensionality that are orders of magnitude larger. Hence, despite general merits of deep learning, the promise here is confronted by formidable challenges both practical and epistemological.…”

Section: Dynamics Of Hysteresismentioning

confidence: 99%

“…With analogies between animal learning [6][7][8][55][56][57][380][381][382] and machine learning [49][50][51][52][53][54]288,291,[383][384][385][386][387][388][389][390][391][392][393][394][395][396], the theory of a mixture of experts is based on advantages of modular parallelism and conditional computation for balancing versatility and efficiency in optimal control. As with the mixture-of-experts (MoE) architecture per se (which has also proven effective for sparse scaling of a deep neural network), the scope of this consilient theory can be extended to systems of varying levels of expertise as well as nonexpert controllers and their numerous choice and action biases (cf.…”

Section: The Optimality Of Nonexpert Control With Lessons For ML and Aimentioning

confidence: 99%

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Colas,

O’Doherty,

Grafton

2024

PLoS Comput Biol

View full text Add to dashboard Cite

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants—even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

show abstract

“…These include approaches based on SMT-solving [15], [27], [30]- [32], [35], approaches based on reachability analysis and abstract interpretation [21], [40], [56], [63], and many others. Recently, the verification of DRL-trained DNNs and the systems in which they operate also received some attention [5], [9], [19], [33], [65]. Our work here is another step toward applying DNN verification techniques to real systems of interest.…”

Section: Related Workmentioning

confidence: 99%

Verifying Learning-Based Robotic Navigation Systems

Amir¹,

Corsi²,

Yerushalmi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) has become a dominant deep-learning paradigm for various tasks in which complex policies are learned within reactive systems. In parallel, there has recently been significant research on verifying deep neural networks. However, to date, there has been little work demonstrating the use of modern verification tools on real, DRLcontrolled systems. In this case-study paper, we attempt to begin bridging this gap, and focus on the important task of mapless robotic navigation -a classic robotics problem, in which a robot, usually controlled by a DRL agent, needs to efficiently and safely navigate through an unknown arena towards a desired target. We demonstrate how modern verification engines can be used for effective model selection, i.e., the process of selecting the best available policy for the robot in question from a pool of candidate policies. Specifically, we use verification to detect and rule out policies that may demonstrate suboptimal behavior, such as collisions and infinite loops. We also apply verification to identify models with overly conservative behavior, thus allowing users to choose superior policies that are better at finding an optimal, shorter path to a target. To validate our work, we conducted extensive experiments on an actual robot, and confirmed that the suboptimal policies detected by our method were indeed flawed. We also compared our verification-driven approach to state-of-the-art gradient attacks, and our results demonstrate that gradient-based methods are inadequate in this setting. Our work is the first to demonstrate the use of DNN verification backends for recognizing suboptimal DRL policies in real-world robots, and for filtering out unwanted policies. We believe that the methods presented in this work can be applied to a large range of application domains that incorporate deep-learning-based agents.

show abstract

MoËT: Mixture of Expert Trees and its application to verifiable reinforcement learning

Cited by 19 publications

References 25 publications

Revealing principles of autonomous thermal soaring in windy conditions using vulture-inspired deep reinforcement-learning

Revealing principles of autonomous thermal soaring in windy conditions using vulture-inspired deep reinforcement-learning

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Verifying Learning-Based Robotic Navigation Systems

Contact Info

Product

Resources

About