No abstract

As classical methods are intractable for solving Markov decision processes (MDPs) requiring a large state space, decomposition and aggregation techniques are very useful to cope with large problems. These techniques are in general a special case of the classic Divide-and-Conquer framework to split a large, unwieldy problem into smaller components and solving the parts in order to construct the global solution. This paper reviews most of decomposition approaches encountered in the associated literature over the past two decades, weighing their pros and cons. We consider several categories of MDPs (average, discounted, and weighted MDPs), and we present briefly a variety of methodologies to find or approximate optimal strategies.

I n t r o d u c t i o nIn this paper we consider a singularly perturbed Markov Decision Process with the limiting average cost criterion. We assume that the underlying process is composed of n separate irreducible processes, and that the small perturbation is such that it "unites" these processes into a single irreducible process. This structure corresponds to the Markov chains admitting "strong and weak interactions" that arises in many applications, and was studied by a number of authors (e.g., see Delebecque and Quadrat [5] In Section 2 we introduce the formulation and some results given by Bielecki and Fi1a.r [2] of the underlying control problem for the singula.rly perturbed MDP; the so-called "limit Markov Control Problem'' (limit MCP). In particular these authors proved that an optimal solution to the perturbed MDP can be approximated by an optimal solution of the limit MCP for sufficiently small perturbation.In Sectioii 3 we demonstrate tha.t the above limit Markov Control Problem ca.n be solved by a suitably constructed linear program.In Section 4 we construct an algorithm for solving the limit Markov Control Problem based on the policy improvement method. Recently we learned that this algorithm is similar to one given by Pervoewnskii and Gaitsgori [12]. However, these authors did not explicitly consider the limit Markov Control Problem, and worked only in t,he smaller class of deterministic strategies. With every A E IT we shall associate the following quantities: Definitions and Preliminaries. ( The "classical" limiting average Markov Decision problem is the optimization problem: Find x 0 E lI such that J(s,aO) = m;xJ(s,n) for dl s E S.A stra.tegy K O satisfying the above will be called optimal. It is well known that there always exists an optimal deterministic policy and there is a number of finite algorithms for its computation (e.g., Dena.rdo [6], Derman (71, Kallenberg [8]).In this paper we shall assume that:( A l ) S = u:,,S; where Si n Sj = 8 if i # j, n > I, cards; = n;, nl + . . . Consequently we can think of r as being the "union" of n smaller MDP's r;, defined on the state space Si, for each i = 1 , 2 , . , . ,n, respectively. Note that if IIi is the space of stationary strategies in r;, then a strategy A E IT in I? can be written in the natural way as K = (K', r2,. . . , +' ), where xi E IIi. The probability transition mat,rix in ri corresponding to ri is, of course, defined by: P;(ri) := ( p # +~( x~) ) # ,~,~~, , and the generator G;(n') and the Cesaro-limit P;(ri) matrices can be defined in a manner analogous to that in the original process I?. In addition, we assume that:'Not? l.hat action z E A ( s ) may not be the same as action i E A(s') if "'The nointion [ u / ,~ will be tlsed to denote the s-th entry of a vector U s # SI. This simplification of notal.ion shoold not cause ambiguity.

Polymer flooding is one of the most mature enhanced oil recovery (EOR) methods with many field implementations including those in China, Germany, Oman, and USA. The primary role of polymer is increasing the injected water viscosity, hence reducing the displacing front mobility and thereby improving the macroscopic sweep efficiency. Polyacrylamide, the main polymer used in EOR applications, achieve this increase in viscosity due to the large molecular size of its chains as well as the ionic repulsion between the chains. Those same properties result in significant interactions between the transported polymer molecules and the porous medium, including adsorption, mechanical entrapment, and hydrodynamic retention. Those phenomena, in turn, can lead to polymer losses, injectivity reductions and inaccessible pore volumes. Despite the maturity of polymer flooding, few implementations and research studies have targeted carbonates. Thus, a clear understanding of the magnitude and significance of those interactions and effects for carbonates is lacking. Those phenomena are critical for both numerical predictions and actual performances of polymer flood. Therefore, in this work we investigate thoroughly polymer losses, injectivity reductions and inaccessible pore volumes for a slightly viscous Arabian carbonate reservoir that exhibits high salinity and high temperature conditions. For this purpose, we perform single phase displacement experiments at reservoir conditions. Representative materials were used including simulated brines reflecting connate and injection brine salinities, dead crude oil, and aged reservoir plugs. Core plugs with a wide permeability range from 45.2 md to 12836 md were used for the tests. A pre-screened polyacrylamide was used at an injection concentration of 5,500 ppm. A 2,000 ppm tracer was added into the polymer solution to assess polymer interactions. The effluent polymer concentrations were determined by total organic carbon (TOC) method, and tracer concentrations were analyzed by gas chromatography (GC). Results showed that resistance factor (RF) tended to be higher for tighter samples. RF increased with increasing injection rate for lower permeability samples and decreased with increasing injection rate for higher permeability samples. Residual resistance factor (RRF) slightly decreased with increasing injection rate. RRF correlated well with pore size, with larger pore size corresponding to lower RRF. The effective in-situ viscosity of the polymer was constant at lower injection rates. However, at higher rates, the effective in-situ viscosity increased with injection, exhibiting a shear thickening behavior. Moreover, the polymer exhibited dynamic retention ranging from 0.155 to 0.530 mg/g-rock, and showed a decreasing trend for more permeable core sample. Finally, the studied carbonate constituted of 15.2% to 20.9% pore-volume that was inaccessible to the polymer. Those results besides being essential for numerical-based upscaling of polymer flooding, shed light on some of the similarities and differences between sandstones and carbonates when it comes to chemical EOR application.