Asynchronous Stochastic Approximation Based Learning Algorithms for As-You-Go Deployment of Wireless Relay Networks Along a Line

Abstract

We are motivated by the need, in emergency situations, for impromptu (or "as-you-go") deployment of multihop wireless networks, by human agents or robots (e.g., unmanned aerial vehicles (UAVs)); the agent moves along a line, makes wireless link quality measurements at regular intervals, and makes on-line placement decisions using these measurements. As a first step we have formulated such deployment along a line as a sequential decision problem. In our earlier work, reported in [1], we proposed two possible deployment approaches: (i) the pure as-you-go approach where the deployment agent can only move forward, and (ii) the explore-forward approach where the deployment agent explores a few successive steps and then selects the best relay placement location among them. The latter was shown to provide better performance (in terms of network cost, network performance and power expenditure), but at the expense of more measurements and deployment time, which makes explore-forward impractical for quick deployment by an energy constrained agent such as a UAV. Further, since in emergency situations the terrain would be unknown, the deployment algorithm should not require a-priori knowledge of the parameters of the wireless propagation model. In [1] we, therefore, developed learning algorithms for the explore-forward approach.The current paper fills in an important gap by providing deploy-and-learn algorithms for the pure as-you-go approach. We formulate the sequential relay deployment problem as an average cost Markov decision process (MDP), which trades off among power consumption, link outage probabilities, and the number of relay nodes in the deployed network. While the pure as-you-go deployment problem was previously formulated as a discounted cost MDP (see [1]), the discounted cost MDP formulation was not amenable for learning algorithms that are proposed in this paper. In this paper, first we show structural results for the optimal policy corresponding to the average cost MDP, and provide new insights into the optimal policy. Next, by exploiting the special structure of the average cost optimality equation and by using the theory of asynchronous stochastic approximation (in single and two timescale), we develop two learning algorithms that asymptotically converge to the set of optimal policies as deployment progresses. Numerical results show reasonably fast speed of convergence, and hence the model-free algorithms can be useful for practical, fast deployment of emergency wireless networks.Index Terms-Wireless networks, impromptu network deployment, as-you-go relay placement, relay placement by UAV, Markov decision process, stochastic approximation. !

Autonomous Robot Path Construction Prototype Using Wireless Sensor Networks

“…There has been prior work on the problem of "as-you-go" or impromptu deployment of a WSN. In [4] and [7] are proposed algorithms for "as-you-go" deployment, in which an agent, human or robot, measures link quality at equally spaced intervals and makes placement decisions, while only moving forward along a line. Some works consider a purely autonomous deployment of a WSN by mobile robots.…”

“…Cooperation between multiple agents is seen as an important problem to be explored. In [4] and [7] it is pointed the need for algorithms where two or more robots cooperate to deploy a WSN. While [6] calls for field test using two mobile robots and to address the communication issues that arise from coordination between multiple robots.…”

“…The need for extensible WSNs is tied with monitoring situations that require the deployment of a WSN. Situations such as forest fires or fires is large buildings [4], determining the extent of damage of earthquakes [5], nuclear disasters as in Fukushima, Japan, rescue operations, exploratory operations, mapping flooded caves, battlefield reconnaissance or planetary exploration [6]. In some of these situations it is possible for a human operator to deploy the nodes of the WSN.…”

Intelligent UAV Deployment for a Disaster-Resilient Wireless Network

“…In general, all UAV ABSs are connected to each other and to a central control station [5][6][7][8][10][11][12][13][14][15]. The works in the literature consider both centralized [5,6,[16][17][18][19][20] and decentralized approaches [8,[21][22][23] for the deployment and control of UAV networks. Centralized algorithms are capable of providing highly accurate decisions compared to the decentralized approaches.…”

“…In addition, being different to our work, the impact of LoS is not considered in [9,25,26,30] for small-scale fading, which can significantly affect the performance in dense urban environments. Moreover, the altitude of the UAVs are not taken into account in [19,27], and a fixed UAV altitude assumption is considered in [5]. It is well known that the altitude of the UAV plays a major role in the performance of the network, due to its impact on the path loss, coverage radius, and the probability of line of sight (PLoS).…”