We present for the first time an asymptotic convergence analysis of two time-scale stochastic approximation driven by 'controlled' Markov noise. In particular, both the faster and slower recursions have non-additive controlled Markov noise components in addition to martingale difference noise. We analyze the asymptotic behavior of our framework by relating it to limiting differential inclusions in both time-scales that are defined in terms of the ergodic occupation measures associated with the controlled Markov processes. Finally, we present a solution to the off-policy convergence problem for temporal difference learning with linear function approximation, using our results.

No abstract

No abstract

We study a many-server queueing model with server vacations, where the population size dynamics of servers and customers are coupled: a server may leave for vacation only when no customers await, and the capacity available to customers is directly affected by the number of servers on vacation. We focus on scaling regimes in which server dynamics and queue dynamics fluctuate at matching time scales so that their limiting dynamics are coupled. Specifically, we argue that interesting coupled dynamics occur in (a) the Halfin–Whitt regime, (b) the nondegenerate slowdown regime, and (c) the intermediate near Halfin–Whitt regime, whereas the dynamics asymptotically decouple in the other heavy-traffic regimes. We characterize the limiting dynamics, which are different for each scaling regime. We consider relevant respective performance measures for regimes (a) and (b)—namely, the probability of wait and the slowdown. Although closed-form formulas for these performance measures have been derived for models that do not accommodate server vacations, it is difficult to obtain closed-form formulas for these performance measures in the setting with server vacations. Instead, we propose formulas that approximate these performance measures and depend on the steady-state mean number of available servers and previously derived formulas for models without server vacations. We test the accuracy of these formulas numerically.