Resource provisioning and profit maximization for transcoding in Information Centric Networking

Abstract

Abstract-Adaptive bitrate streaming (ABR) has been widely adopted to support video streaming services over heterogeneous devices and varying network conditions. With ABR, each video content is transcoded into multiple representations in different bitrates and resolutions. However, video transcoding is computing intensive, which requires the transcoding service providers to deploy a large number of servers for transcoding the video contents published by the content producers. As such, a natural question for the transcoding service provider is how to provision the computing resource for transcoding the video contents while maximizing service profit. To address this problem, we design a cloud video transcoding system by taking the advantage of cloud computing technology to elastically allocate computing resource. We propose a method for jointly considering the task scheduling and resource provisioning problem in two timescales, and formulate the service profit maximization as a two-timescale stochastic optimization problem. We derive some approximate policies for the task scheduling and resource provisioning. Based on our proposed methods, we implement our open source cloud video transcoding system Morph and evaluate its performance in a real environment. The experiment results demonstrate that our proposed method can reduce the resource consumption and achieve a higher profit compared with the baseline schemes.

Real-world Machine Learning Systems: A survey from a Data-Oriented Architecture Perspective

“…Data that is produced by different sources in a continuous fashion fits the streams data model according to our review. That is the case of works that process video streams [43,49,56,60], sensors data [14,23,31,36,59,75,83,106,112,126], social media data [22,81,109,123,125], and network metrics [102,116]. Databases schemes are more appropriate as shared data models when system components require to store historical records [9,29,57] or process large amounts of data [105].…”

“…The Apache Spark machine learning library (MLlib) is used to analyse such data in a distributed fashion. Systems that partially adopt data coupling (i.e., 13%) [10,22,49,56,81,112] are the ones where some components communicate through data mediums and others using traditional process calls (e.g., API calls or Remote Procedure Calls (RPC)). These design decisions are supported by particular requirements these systems need to satisfy.…”

“…These design decisions are supported by particular requirements these systems need to satisfy. For example, systems can use API calls to collect data from data sources [10,81] or to receive requests from and send responses to end users [49] reviewed papers (i.e., 56.5%) do not adopt data coupling as data is hidden by each systems' components and these components interact using traditional process calls. They can use API calls [14,31,48,55,62,67,75,86,99,102,109] or remote procedure calls (RPC) [8,15,16,29,43,47,57,58,98,101,103,111,123,126] for communication.…”

“…Most of the reviewed papers report ML-based systems that rely on centralised storage (i.e., 50%). Some of these systems are deployed on single cloud servers [10,31,47,49,56,81,98,105], where all data is stored. Other systems are deployed on resource-constraint devices like robots [57,67,106], or small processing units (e.g., Raspberry Pis) [23,43,62].…”

“…Learning models are deployed on distributed computing nodes that process requests, make decisions, and provide systems' functionalities. This approach also proves popular for big data pre-and postprocessing pipelines [33,49,66,72]. In most cases these nodes provide such functionalities without any direct cooperation between them [15,60,62,66,99,103,111,126].…”