Reinforcement Learning Methods for Computation Offloading: A Systematic Review

Abstract

Today, cloud computation offloading may not be an appropriate solution for delay-sensitive applications due to the long distance between end-devices and remote datacenters. In addition, offloading to a remote cloud can consume bandwidth and dramatically increase costs. On the other hand, end-devices such as sensors, cameras, and smartphones have limited computing and storage capacity. Processing tasks on such battery-powered and energy-constrained devices becomes even more complex. To address these challenges, a new paradigm called Edge Computing (EC) emerged nearly a decade ago to bring computing resources closer to end-devices. Here, edge servers located between the end-device and the remote cloud perform user tasks. Recently, several new computing paradigms such as Mobile Edge Computing (MEC) and Fog Computing (FC) have emerged to complement Cloud Computing (CC) and EC. Although these paradigms are heterogeneous, they can further reduce energy consumption and task response time, especially for delay-sensitive applications. The computation offloading is a multi-objective, NP-hard optimization problem. A significant part of previous research in this field is devoted to Machine Learning (ML) methods. One of the essential types of ML is Reinforcement Learning (RL), in which an agent learns how to make the best decision using the experiences gained from the environment. This paper provides a systematic review of the widely used RL approaches in computation offloading. It covers research in complementary paradigms such as mobile cloud computing (MCC), edge computing, fog computing, and the Internet of Things (IoT). We explain the reasons for using various RL methods in computation offloading from a technical point of view. This analysis includes both binary offloading and partial offloading techniques. For each method, the essential elements of RL and the characteristics of the environment are discussed regarding the most important criteria. Research challenges and Future trends are also mentioned.

Harnessing idle edge resources for execution of cloud services: A comprehensive review

“…15 Meanwhile, other research endeavors have aimed to elucidate the broader application of specific techniques to offloading in general. 14,[16][17][18] The rest of this work is organized as follows. Section 2 describes the methodology used in this review.…”

Trajectory-Aware Offloading Decision in UAV-Aided Edge Computing: A Comprehensive Survey

“…This is the art of offloading decisions, finding a precise balance between control and efficiency. It refers to the strategic decision regarding where computational tasks are processed on the UAV itself (locally) or at the terrestrial base station of the network [ 1 ]. This decision is crucial for optimizing the utilization of resources, reducing latency, and improving the overall system efficiency [ 24 ].…”

“…Internet of Things (IoT) devices and their applications with diverse quality-of-service (QoS) requirements have led to an anomalous demand for computation-intensive and latency-sensitive tasks, including real-time online gaming, image or video processing, autonomous vehicles, augmented reality (AR), and virtual reality (VR) [ 1 ]. However, IoT users often face limitations owing to resource constraints, including limited computational resources and energy, making it challenging to execute related applications effectively [ 2 ].…”

“…Nguyen et al [ 29 ] summarized the RL-based offloading techniques for aerial computing. In [ 1 ], the authors systematically reviewed RL techniques used for computation offloading in the overall edge computing. These studies primarily focused on terrestrial edge servers.…”

Road Intersection Extraction Based on Low-Frequency Vehicle Trajectory Data

“…A vehicle's GNSS trajectory is a record of the vehicle's path, containing rich road information (e.g., lanes, turns, speed limits, road widths, and road intersections) that directly reflects the road network's geometric characteristics and provides a new database for road intersection extraction [8,9]. Therefore, an increasing number of scholars are utilizing vehicle GNSS trajectory data in tandem with machine learning to extract road geometry data and examine vehicle behavior, among other applications [10][11][12][13][14][15][16] The traditional road intersection detection algorithm takes the vehicle trajectory's unique turning information and speed information at the road intersection as the benchmark, extracts the turning points after ensuring the intersection's accuracy, and then extracts the road intersection on the basis of the turning points' clustering. Qixing developed a scale-and orientation-invariant traj-SIFT feature to localize and recognize junctions using a supervised learning framework [17].…”