Tiyuntsong: A Self-Play Reinforcement Learning Approach for ABR Video Streaming

Abstract

Existing reinforcement learning (RL)-based adaptive bitrate (ABR) approaches outperform the previous fixed control rules based methods by improving the Quality of Experience (QoE) score, as the QoE metric can hardly provide clear guidance for optimization, finally resulting in the unexpected strategies. In this paper, we propose Tiyuntsong, a selfplay reinforcement learning approach with generative adversarial network (GAN)-based method for ABR video streaming. Tiyuntsong learns strategies automatically by training two agents who are competing against each other. Note that the competition results are determined by a set of rules rather than a numerical QoE score that allows clearer optimization objectives. Meanwhile, we propose GAN Enhancement Module to extract hidden features from the past status for preserving the information without the limitations of sequence lengths. Using testbed experiments, we show that the utilization of GAN significantly improves the Tiyuntsong's performance. By comparing the performance of ABRs, we observe that Tiyuntsong also betters existing ABR algorithms in the underlying metrics.Index Terms-Adaptive Bitrate Streaming, Self-play Reinforcement learning 1 Tinyuntsong: Also named as Cloud Ascending Ladder, a qinggong skill in the Chinese wuxia novel The Heaven Sword and Dragon Saber by Jin Yong. The skill enables the user to travel at high speeds and leap to extreme heights by stepping one foot on the other one.

Self-play reinforcement learning for video transmission

“…Note that the output is an n-dim vector indicating the probability of the bitrate being selected under the current ABR state S k . In this work, we set n = 6, which is widely used in ABR papers [9,14].…”

“…Taking a look from another perspective, we observe that the aforementioned problem can be naturally described as a deterministic goal or requirement [9]. For instance, in the most cases, the goal of the adaptive bitrate (ABR) streaming algorithm is to achieve lower rebuffering time first, and next, reaching higher bitrate [10,13].…”

Comyco: Quality-Aware Adaptive Video Streaming via Imitation Learning

“…Nevertheless, such methods are built in pre-assumptions that it is hard to keep its performance in all considered network scenarios [28]. To this end, learning-based ABR algorithms [15,18,28] are proposed to solve the problem from another perspective: it Figure 1: We evaluate quality-aware ABR algorithm and bitrate-aware ABR algorithm with the same video on Norway network traces respectively. Results are plotted as the curves of selected bitrate, buffer occupancy and the selected chunk's VMAF ( §5.1, [38]) for entire sessions.…”

“…Recall that the key principle of RL-based method is to maximize reward of each action taken by the agent in given states per step, since the agent doesn't really know the optimal strategy [45]. However, recent work [6,18,28,36,43,51] has demonstrated that the ABR process can be precisely emulated by an offline virtual player ( §6.1) with complete future network information. What's more, by taking several steps ahead, we can further accurately estimate the near-optimal expert policy of any ABR state within an acceptable time ( §4.2).…”

“…D-DASH [15] uses Deep Q-learning method to perform a comprehensive evaluation based on state-of-the-art algorithms. Tiyuntsong optimizes itself towards a rule or a specific reward via the competition with two agents under the same network condition [18]. Imitation Learning meets Networking.…”