Experimental Demonstration of Multi-Agent Learning and Planning under Uncertainty for Persistent Missions with Automated Battery Management

2014 American Control Conference

et al. 2014

Self Cite

This paper introduces a novel hierarchical decomposition approach for solving Multiagent Markov Decision Processes (MMDPs) by exploiting coupling relationships in the reward function. MMDP is a natural framework for solving stochastic multi-stage multiagent decision-making problems, such as optimizing mission performance of Unmanned Aerial Vehicles (UAVs) with stochastic health dynamics. However, computing the optimal solutions is often intractable because the state-action spaces scale exponentially with the number of agents. Approximate solution techniques do exist, but they typically rely on extensive domain knowledge. This paper presents the Hierarchically Decomposed MMDP (HD-MMDP) algorithm, which autonomously identifies different degrees of coupling in the reward function and decomposes the MMDP into a hierarchy of smaller MDPs that can be solved separately. Solutions to the smaller MDPs are embedded in an autonomously constructed tree structure to generate an approximate solution to the original problem. Simulation results show HD-MMDP obtains more cumulative reward than that of the existing algorithm for a ten-agent Persistent Search and Track (PST) mission, which is a cooperative multi-UAV mission with more than 10 19 states, stochastic fuel consumption model, and health progression model.

“…dent trails. The GA-Dec-MMDP algorithm was previously shown to be an effective planner for the PST problem [18] and Fig. 3 shows that HD-MMDP outperforms GA-Dec-MMDP by more than 35%.…”

Section: A Persistent Search and Track (Pst)mentioning

confidence: 86%

Section: A Persistent Search and Track (Pst)mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Planning for large-scale multiagent problems via hierarchical decomposition with applications to UAV health management

Chen

2014 American Control Conference

et al. 2014

Self Cite

“…This paper presents the details of the design and development of an automated battery change/recharge station that satisfies these requirements and enables long duration autonomous missions. This new capability is highlighted through flight-test results of planning and learning algorithms [14], [15] for a persistent surveillance and tracking mission.…”

Section: Introductionmentioning

confidence: 99%

An Automated Battery Management System to Enable Persistent Missions With Multiple Aerial Vehicles

IEEE/ASME Trans. Mechatron.

Toksoz

et al. 2015

Self Cite

119

This paper presents the development and hardware implementation of an autonomous battery maintenance mechatronic system that significantly extends the operational time of battery powered small-scaled unmanned aerial vehicles (UAVs). A simultaneous change and charge approach is used to overcome the significant downtime experienced by existing charge-only approaches. The automated system quickly swaps a depleted battery of a UAV with a replenished one while simultaneously recharging several other batteries. This results in a battery maintenance system with low UAV downtime, arbitrarily extensible operation time, and a compact footprint. Hence, the system can enable multiagent UAV missions that require persistent presence. This capability is illustrated by developing and testing in flight a centralized autonomous planning and learning algorithm that incorporates a probabilistic health model dependent on vehicle battery health that is updated during the mission, and replans to improve the performance based on the improved model. Flight test results are presented for a 3-h-long persistent mission with three UAVs that each has an endurance of 8-10 min on a single battery charge (more than 100 battery swaps).Index Terms-Battery management systems, learning (artificial intelligence), Markov processes, multiagent systems, unmanned aerial vehicles (UAVs).

Distributed Learning for Planning Under Uncertainty Problems with Heterogeneous Teams

Chen

et al. 2013

J Intell Robot Syst

Self Cite

This paper considers the problem of multiagent sequential decision making under uncertainty and incomplete knowledge of the state transition model. A distributed learning framework, where each agent learns an individual model and shares the results with the team, is proposed. The challenges associated with this approach include choosing the model representation for each agent and how to effectively share these representations under limited communication. A decentralized extension of the model learning scheme based on the Incremental Feature Dependency Discovery (Dec-iFDD) is presented to address the distributed learning problem. The representation selection problem is solved by leveraging iFDD's property of adjusting the model complexity based on the observed data. The model sharing problem is addressed by having each agent rank the features of their representation based on the model reduction error and broadcast the most relevant features to their teammates. The algorithm is tested on the multiagent block building and the persistent search and track missions. The results show that the proposed distributed learning scheme is particularly useful in heterogeneous learning setting, where each agent learns significantly different models. We show through large-scale planning under uncertainty simulations and flight experiments with state-dependent actuator and fuel-burn-rate uncertainty that our planning approach can outperform planners that do not account for heterogeneity between agents.