Deep Q-Network-Based Cooperative Transmission Joint Strategy Optimization Algorithm for Energy Harvesting-Powered Underwater Acoustic Sensor Networks

Han, Song; Li, Luo; Li, Xinbin

doi:10.3390/s20226519

Cited by 14 publications

(14 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For implementation RL based relay selection in UWA sensor networks, Su et al extended the scheme [29] to internet of underwater things (IoUT) network [30] with additional power adjust strategy to maximize the end-to-end signal to noise ratio (SNR) of the system. Han et al [31] investigated reinforcement learning based joint relay selection and power allocation in energy harvesting UWA cooperative networks. In the proposed model, a joint state expression is presented to better reveal the relationship between learning and environment, and a reward function that consists of channel capacity and energy consumption is proposed for adjusting power strategy to balance instantaneous capacity and long-term quality of service (QoS).…”

Section: Related Workmentioning

confidence: 99%

“…The relay adjusts transmission power by the battery level, harvested energy and CSI to maximize the cumulative rate of the UWA network. In [30][31][32], the UWA channel is simulated by integrated empirical formula and statistic model, and the relay selection criteria is similar to wireless sensor networks, mainly by CSI. This paper will systemically analyze how the specific phenomena of UWA affect relay selection criteria in Section 1.2, will design an effective state and reward expression for UWA RL based relay selection with the consideration of UWA propagation delay in Section 3.1.…”

Section: Related Workmentioning

confidence: 99%

“…In the above RL based UWA cooperation systems [28][29][30][31][32], the learning related parameters are static. We consider that the learning process is a dynamic process to explore the possible actions for maximizing the reward.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks

Zhang

Shen

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

In the complex and dynamically varying underwater acoustic (UWA) channel, cooperative communication can improve throughput for UWA sensor networks. In this paper, we design a reasonable relay selection strategy for efficient cooperation with reinforcement learning (RL), considering the characteristics of UWA channel variation and long transmission delay. The proposed scheme establishes effective state and reward expression to better reveal the relationship between RL and UWA environment. Meanwhile, simulated annealing (SA) algorithm is integrated with RL to improve the performance of relay selection, where exploration rate of RL is dynamically adapted by SA optimization through the temperature decline rate. Furthermore, the fast reinforcement learning (FRL) strategy with pre-training process is proposed for practical UWA network implementation. The whole proposed SA-FRL scheme has been evaluated by both simulation and experimental data. The simulation and experimental results show that the proposed relay selection scheme can converge more quickly than classical RL and random selection with the increase of the number of iterations. The reward, access delay and data rate of SA-FRL can converge at the highest value and are close to the ideal optimum value. All in all, the proposed SA-FRL relay selection scheme can improve the communication efficiency through the selection of the relay nodes with high link quality and low access delay.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks

Zhang

Shen

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…[ 1 , 2 , 3 ]. Compared with terrestrial wireless sensor networks, the transmission in UASNs suffers from low data rates due to large propagation attenuation, limited bandwidth, and time-varying channels [ 4 ]. Generally, the transmission data rate highly depends on the selection of modulation method, coding rate, and transmission power, which will be referred to as transmission configuration in this paper.…”

Section: Introductionmentioning

confidence: 99%

Double-Scale Adaptive Transmission in Time-Varying Channel for Underwater Acoustic Sensor Networks

Cen

Liu

et al. 2021

Sensors

View full text Add to dashboard Cite

The communication channel in underwater acoustic sensor networks (UASNs) is time-varying due to the dynamic environmental factors, such as ocean current, wind speed, and temperature profile. Generally, these phenomena occur with a certain regularity, resulting in a similar variation pattern inherited in the communication channels. Based on these observations, the energy efficiency of data transmission can be improved by controlling the modulation method, coding rate, and transmission power according to the channel dynamics. Given the limited computational capacity and energy in underwater nodes, we propose a double-scale adaptive transmission mechanism for the UASNs, where the transmission configuration will be determined by the predicted channel states adaptively. In particular, the historical channel state series will first be decomposed into large-scale and small-scale series and then be predicted by a novel k-nearest neighbor search algorithm with sliding window. Next, an energy-efficient transmission algorithm is designed to solve the problem of long-term modulation and coding optimization. In particular, a quantitative model is constructed to describe the relationship between data transmission and the buffer threshold used in this mechanism, which can then analyze the influence of buffer threshold under different channel states or data arrival rates theoretically. Finally, numerical simulations are conducted to verify the proposed schemes, and results show that they can achieve good performance in terms of channel prediction and energy consumption with moderate buffer length.

show abstract

“…Different from terrestrial radio-frequency wireless communication, the use of acoustic signal seems to be a better way in OSNs [ 8 ]. Unfortunately, it is difficult to estimate a target with the acoustic signal due to various adverse factors, including the dynamics of the environment, multipath and absorption loss of the signal, and noise interference [ 7 , 9 , 10 ]. In this context, how to locate a target in such a complex ocean environment is a challenge.…”

Section: Introductionmentioning

confidence: 99%

Two-Phase Robust Target Localization in Ocean Sensor Networks Using Received Signal Strength Measurements

Zhang

Mei

et al. 2021

Sensors

View full text Add to dashboard Cite

Target localization plays a vital role in ocean sensor networks (OSNs), in which accurate position information is not only a critical need of ocean observation but a necessary condition for the implementation of ocean engineering. Compared with other range-based localization technologies in OSNs, the received signal strength (RSS)-based localization technique has attracted widespread attention due to its low cost and synchronization-free nature. However, maintaining relatively good accuracy in an environment as dynamic and complex as the ocean remains challenging. One of the most damaging factors that degrade the localization accuracy is the uncertainty in transmission power. Besides the equipment loss, the uncertain factors in the fickle ocean environment may result in a significant deviation between the standard rated transmission power and the usable transmission power. The difference between the rated and actual transmission power would lead to an extra error when it comes to the localization in OSNs. In this case, a method that can locate the target without needing prior knowledge of the transmission power is proposed. The method relies on a two-phase procedure in which the location information and the transmission power are jointly estimated. First, the original nonconvex localization problem is transformed into an alternating non-negativity-constrained least square framework with the unknown transmission power (UT-ANLS). Under this framework, a two-stage optimization method based on interior point method (IPM) and majorization-minimization tactic (MMT) is proposed to search for the optimal solution. In the first stage, the barrier function method is used to limit the optimization scope to find an approximate solution to the problem. However, it is infeasible to approach the constraint boundary due to its intrinsic error. Then, in the second stage, the original objective is converted into a surrogate function consisting of a convex quadratic and concave term. The solution obtained by IPM is considered the initial guess of MMT to jointly estimate both the location and transmission power in the iteration. In addition, in order to evaluate the performance of IPM-MM, the Cramer Rao lower bound (CRLB) is derived. Numerical simulation results demonstrate that IPM-MM achieves better performance than the others in different scenarios.

show abstract

Deep Q-Network-Based Cooperative Transmission Joint Strategy Optimization Algorithm for Energy Harvesting-Powered Underwater Acoustic Sensor Networks

Cited by 14 publications

References 33 publications

Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks

Reinforcement Learning Based Relay Selection for Underwater Acoustic Cooperative Networks

Double-Scale Adaptive Transmission in Time-Varying Channel for Underwater Acoustic Sensor Networks

Two-Phase Robust Target Localization in Ocean Sensor Networks Using Received Signal Strength Measurements

Contact Info

Product

Resources

About