MPC-based Reinforcement Learning for Economic Problems with Application to Battery Storage

Multi-agent Battery Storage Management using MPC-based Reinforcement Learning

“…A very low or very high state of charge decrease the battery lifetime [18]. Note that under optimality condition, the buying and selling variables can not be non-zero at the same time, then the difference of buying and selling b i k − s i k can be considered as the input of the system ( [11]). The closed-loop performance J reads as:…”

“…The input a k is selected according to the corresponding parametric policy π θ in (11) with possible addition of small random exploration.…”

“…where θ i b , θ i s , φ i 1,2,3 , and T i 1,2,3 are among the adjustable parameters θ, i.e: θ = {θ 1,...,n α , θ 1,...,n δ , θ 1,...,n b , θ 1,...,n s , φ 1,...,n 1,2,3 , T 1,...,n 1,2,3 }. (24) One can use more generic function approximators in (23), however, in [11], it shows that, for this kind of battery storage problem, quadratic parameterizations for the stage and terminal costs in the MPC-based policy approximator are rich enough to capture the optimal policy.…”

“…Figure 7 (Left) shows the learned policy for each agent after the last RL-step. From the previous work ( [11]), we know that the linear economic stage cost often yields a (nearly) bangbang structure optimal policy when the battery dynamics are stochastic and linear. This figure demonstrates the similar optimal policy consequence as expected.…”

“…Recent researches have developed further the combination of RL and MPC (see e.g. [11], [12], [13], [14], [15]).…”