Flow Control as a Stochastic Optimal Control Problem with Incomplete Information

Abstract

A nonlinear stochastic control problem related to flow control is considered. It is assumed that the state of a link is described by a controlled hidden Markov process with a finite state set, while the loss flow is described by a counting process with intensity depending on a current transmission rate and an unobserved link state. The control is the transmission rate, and it has to be chosen as a nonanticipating process depending on the observation of the loss process. The aim of the control is to achieve the maximum of some utility function that takes into account losses of the transmitted information. Originally, the problem belongs to the class of stochastic control problems with incomplete information; however, optimal filtering equations that provide estimation of the current link state based on observations of the loss process allow one to reduce the problem to a standard stochastic control problem with full observations. Then a necessary optimality condition is derived in the form of a stochastic maximum principle, which allows us to obtain explicit analytic expressions for the optimal control in some particular cases. Optimal and suboptimal controls are investigated and compared with the flow control schemes used in TCP/IP (Transmission Control Protocols/Internet Protocols) networks. In particular, the optimal control demonstrates a much smoother behavior than the TCP/IP congestion control currently used. 0032-9460/05/4102-0150 c 2005 Pleiades Publishing, Inc.

Analysis and Filtration of Special Discrete-Time Markov Processes. II. Optimal Filtration

“…Figures 2-4 show the results of estimation of the component Y t by our algorithm, by the standard TCP algorithm, and with the use of information on the loss process. Figure 5 shows the results of detection of "uncongested" channel state (e 1 + e 2 ) * θ t generated by our algorithm and the algorithm of [2]. Figure 6 shows similar results of detection of "congested" states.…”

“…Results of optimal estimation are compared with those generated by the standard TCP algorithm [3] and time-varying Poisson model [2].…”

“…In the TCP protocol, RTT is estimated by a standard moving-mean algorithm In the time-varying Poisson model of [2], loss intensity jumps, depending on the state of the Markov chain describing the channel state. The state of the process (θ * t , Y t ) * is estimated only with information on the instants of loss of packets without any regard for observations on RTT.…”

“…Estimation of the current parameters of the data transfer channel is reduced to optimal filtration of the states of a special Markov process, and its solution is given in Section 2. Optimal estimates are compared with the results generated by a standard estimation algorithm realized in modern TCP versions and with the estimate given in [2].…”

Congestion Control Framework for Ad-Hoc Wireless Networks

“…This paper deals with the congestion control through the adaptation of the transmission rate. The basic idea of congestion control through transmission rate adaptation has attracted interest and it is studied, from mathematical as well as simulation point of view, in wired networks [26][27][28]. Furthermore, it is argued to be even more beneficial for wireless and ad-hoc networks.…”