Learning-Based Computation Offloading for IoT Devices With Energy Harvesting

Abstract

Internet of Things (IoT) devices can apply mobileedge computing (MEC) and energy harvesting (EH) to provide the satisfactory quality of experiences for computation intensive applications and prolong the battery lifetime. In this article, we investigate the computation offloading for IoT devices with energy harvesting in wireless networks with multiple MEC devices such as base stations and access points, each with different computation resource and radio communication capability. We propose a reinforcement learning based computation offloading framework for an IoT device to choose the MEC device and determine the offloading rate according to the current battery level, the previous radio bandwidth to each MEC device and the predicted amount of the harvested energy. A "hotbooting" Qlearning based computation offloading scheme is proposed for an IoT device to achieve the optimal offloading performance without being aware of the MEC model, the energy consumption and computation latency model. We also propose a fast deep Q-network (DQN) based offloading scheme, which combines the deep learning and hotbooting techniques to accelerate the learning speed of Q-learning. We show that the proposed schemes can achieve the optimal offloading policy after sufficiently long learning time and provide their performance bounds under two typical MEC scenarios. Simulations are performed for IoT devices that use wireless power transfer to capture the ambient radio-frequency signals to charge the IoT batteries. Simulation results show that the fast DQN-based offloading scheme reduces the energy consumption, decreases the computation delay and the task drop ratio, and increases the utility of the IoT device in dynamic MEC, compared with the benchmark Q-learning based offloading.

Applications of Deep Reinforcement Learning in Communications and Networking: A Survey

“…8(d). The authors in [104] aim to design optimal offloading policy for IoT devices with energy harvesting capabilities. The system consists of multiple MEC servers, such as BSs and APs, with different capabilities in computation and communications.…”

“…The IoT device evaluates the reward based on the overall delay, energy consumption, the task drop loss and the data sharing gains in each time slot. Similar to [100], the authors in [104] enhance Q-learning by the hotbooting technique to save the random exploration time at the beginning of learning. The authors also propose a fast DQL offloading scheme that uses hotbooting to initialize the CNN and accelerates the learning speed.…”

Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence

“…More and more research efforts focus on dynamically managing the radio and computational resources for multi-user multi-server edge computing systems, utilizing Lyapunov optimization technologies [18] [19]. In recent years, optimizing computation offloading decisions via DQN is popular [20] [21]. It models the computation offloading problem as a Markov decision process (MBP) and maximize the long-term utility performance.…”

Sensing, Computing, and Communications for Energy Harvesting IoTs: A Survey

“…At edge servers, Xu et al [154] proposed a post-decision state (PDS) based learning algorithm to obtain the optimal workload offloading (to the centralized cloud) policy, which minimizes the long-term system cost. Instead, at the IoT devices, Min et al [155], [156] combined PDS based learning and deep convolutional neural network (CNN) to select the optimal edge device and offloading rate.…”