Low-Rank Value Function Approximation for Co-Optimization of Battery Storage

Cheng, Bolong; Asamov, Tsvetan; Powell, Warren B.

doi:10.1109/tsg.2017.2716382

Cited by 22 publications

(15 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…). Now, instead of having two parameters to optimize over, we have 2T parameters, where T could be in the tens or hundreds of time periods (the largest problem we have worked on exhibited over 40,000 time periods (Cheng et al, 2017)).…”

Section: Value Function Approximationsmentioning

confidence: 99%

Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics

Powell

2022

IEEE Open J. Intell. Transp. Syst.

Self Cite

View full text Add to dashboard Cite

There is a wide range of sequential decision problems in transportation and logistics that require dealing with uncertainty. There are four classes of policies that we can draw on for different types of decisions, but many problems in transportation and logistics will ultimately require some form of direct lookahead policy (DLA) where we optimize decisions over some horizon to make a decision now. The most common strategy is to use a deterministic lookahead (think Google maps), but what if you want to handle uncertainty? In this paper, we identify two major strategies for designing practical, implementable lookahead policies which handle uncertainty in fundamentally different ways. The first is a suitably parameterized deterministic lookahead, where the parameterization is tuned in a stochastic simulator. The second uses an approximate stochastic lookahead, where we identify six classes of approximations, one of which involves designing a "policy-within-a-policy," for which we turn to all four classes of policies. We claim that our approximate lookahead model spans all the classical stochastic optimization tools for lookahead policies, while opening up pathways for new policies. But we also insist that the idea of a parameterized deterministic lookahead is a powerful new idea that offers features that, for some problems, can outperform the more familiar stochastic lookahead policies.

show abstract

Section: Value Function Approximationsmentioning

confidence: 99%

Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics

Powell

2022

IEEE Open J. Intell. Transp. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Other works proposed obtaining first the VF of an MDP and, then, approximate it using a low-rank plus sparse decomposition [20]. On a similar note, a low-rank (rank-one) approximation of the VF has been proposed in the context of energy storage [21], [22]. More recently, [23], [24] proposed to approximate the VF via low-rank optimization in a model-based and off-line setup.…”

Section: Introductionmentioning

confidence: 99%

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

Rozada¹,

Marqués²

2022

Preprint

View full text Add to dashboard Cite

Value-function (VF) approximation is a central problem in Reinforcement Learning (RL). Classical nonparametric VF estimation suffers from the curse of dimensionality. As a result, parsimonious parametric models have been adopted to approximate VFs in high-dimensional spaces, with most efforts being focused on linear and neural-network-based approaches. Differently, this paper puts forth a a parsimonious non-parametric approach, where we use stochastic low-rank algorithms to estimate the VF matrix in an online and model-free fashion. Furthermore, as VFs tend to be multi-dimensional, we propose replacing the classical VF matrix representation with a tensor (multi-way array) representation and, then, use the PARAFAC decomposition to design an online model-free tensor low-rank algorithm. Different versions of the algorithms are proposed, their complexity is analyzed, and their performance is assessed numerically using standardized RL environments.

show abstract

“…In order to further increase the revenue of BESSs, some research work has considered a battery to provide EA and frequency regulation (FR) services simultaneously [18], since FR is a significant income source for energy storage [19][20][21][22][23]. For FR, BESSs are used to regulate the frequency of the power grid by charging or discharging based on the regulation signals sent by the power grid operator [5,19,24,25].…”

Section: Introductionmentioning

confidence: 99%

“…A linear programming method was used to maximize the potential revenue of electrical energy storage from participation in EA and FR in the day-ahead market [27]. Co-optimizing EA and FR services simultaneously is considered a multitimescale problem, and a dynamic programming approach was proposed to solve the co-optimization problem [19,20]. These two existing works on co-optimizing EA and FR services assumed that the electricity prices, regulation signals, or their distributions were known in advance.…”

Section: Introductionmentioning

confidence: 99%

Co-Optimizing Battery Storage for Energy Arbitrage and Frequency Regulation in Real-Time Markets Using Deep Reinforcement Learning

Miao

Chen

et al. 2021

Energies

View full text Add to dashboard Cite

Battery energy storage systems (BESSs) play a critical role in eliminating uncertainties associated with renewable energy generation, to maintain stability and improve flexibility of power networks. In this paper, a BESS is used to provide energy arbitrage (EA) and frequency regulation (FR) services simultaneously to maximize its total revenue within the physical constraints. The EA and FR actions are taken at different timescales. The multitimescale problem is formulated as two nested Markov decision process (MDP) submodels. The problem is a complex decision-making problem with enormous high-dimensional data and uncertainty (e.g., the price of the electricity). Therefore, a novel co-optimization scheme is proposed to handle the multitimescale problem, and also coordinate EA and FR services. A triplet deep deterministic policy gradient with exploration noise decay (TDD–ND) approach is used to obtain the optimal policy at each timescale. Simulations are conducted with real-time electricity prices and regulation signals data from the American PJM regulation market. The simulation results show that the proposed approach performs better than other studied policies in literature.

show abstract

Low-Rank Value Function Approximation for Co-Optimization of Battery Storage

Cited by 22 publications

References 15 publications

Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics

Designing Lookahead Policies for Sequential Decision Problems in Transportation and Logistics

Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning

Co-Optimizing Battery Storage for Energy Arbitrage and Frequency Regulation in Real-Time Markets Using Deep Reinforcement Learning

Contact Info

Product

Resources

About