Reinforcement Learning Based Stochastic Shortest Path Finding in Wireless Sensor Networks

Abstract

Many factors influence the connection states between nodes of wireless sensor networks, such as physical distance, and the network load, making the network's edge length dynamic in abundant scenarios. This dynamic property makes the network essentially form a graph with stochastic edge lengths. In this paper, we study the stochastic shortest path problem on a directional graph with stochastic edge lengths, using reinforcement learning algorithms. we regard each edge length as a random variable following unknown probability distribution and aim to find the stochastic shortest path on this stochastic graph. We evaluate the performance of path-finding algorithms using regret, which represents the cumulative reward difference between the practical path-finding algorithm and the optimal strategy that chooses the global stochastic shortest path every time. We model the path-finding procedure as a Markov decision process and propose two online path-finding algorithms: Q SSP algorithm and SARSA SSP algorithm, both combined with specifically-devised average reward mechanism. We justify the convergence property and correctness of the proposed algorithms theoretically. Experiments conducted on two benchmark graphs illustrate the superior performance of the proposed Q SSP algorithm which outperforms the SARSA SSP algorithm and other competitive algorithms about the regret metric.

MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning

“…If the current policy is used to select actions for such an update, the procedure is called "on-policy learning". For instance, in SARSA method [40], [41], which is an on-policy learning method, the state-action value function is updated as follows…”

“…In most cases, a stochastic (e.g., random) policy is selected as the behaviour policy to ensure enough exploration of new states. One of the most practiced off-policy methods is known as Q-learning [41], [42], [44], [46], which updates the value function using the Bellman optimality equation as follows…”

“…To make the system more stable, only one of the updates shown in Eqs. (41) and (42) will be performed as discussed in [26]. To be more precise, when the size of the covariance is decreasing (i.e., S 1 2 Q π * > 0), the covariances of the RBFs are updated using Eq.…”

“…(42), otherwise, their means are updated using Eqs. (41). Using this approach, unlimited expansion of the RBF covariances is avoided.…”

Energy Optimisation through Path Selection for Underwater Wireless Sensor Networks

“…There are different ways to address this problem, including using genetic algorithm [4], ant colony optimisation [5], [6], using reinforcement learning [7], using nearest neighbour optimisation [8], [9], etc. However, unlike classical shortest path problems, we treat this as a stochastic shortest path problem [10] since the AUV must consider the energy cost of a path in addition to the path length. This is because a shorter path may be more expensive than a longer path if the AUV needs to make many turns for the shorter path.…”