Putting Data Science Pipelines on the Edge

Abstract

This paper proposes a composable "Just in Time Architecture" for Data Science (DS) Pipelines named JITA-4DS and associated resource management techniques for configuring disaggregated data centers (DCs). DCs under our approach are composable based on vertical integration of the application, middleware/operating system, and hardware layers customized dynamically to meet application Service Level Objectives (SLO -application-aware management). Thereby, pipelines utilize a set of flexible building blocks that can be dynamically and automatically assembled and re-assembled to meet the dynamic changes in the workload's SLOs. To assess disaggregated DC's, we study how to model and validate their performance in large-scale settings.

JITA4DS: Disaggregated execution of Data Science Pipelines between the Edge and the Data Centre

“…First, calculate a VDCwide Value of Service (VoS) for a given interval of time, weigh individual values of various instances of pipelines. We have started providing a study on VoS for JITA-4DS [12], we will integrate these observations in further experiments on the current emulation. Second, propose objective functions that can guide heuristics to operate in the large search space of resource configurations.…”

“…To predict the execution time and energy consumption of each application type, we use statistical and data mining techniques [20][21][22][23], which represent the execution time and energy consumption as a function of the VDC resources. A complete study of these aspects for JITA-4DS have been described in [12]. DS pipelines running on top of JITA-4DS VDC's apply sets of big data processing operators to stored data and streams produced by the Internet of Things (IoT) farms (see the upper part of Figure 1).…”

Building Analytics Pipelines for Querying Big Streams and Data Histories with H-STREAM

“…Future work consists of developing a micro-services composition language that can be used for expressing the data processing workflows that can be weaved within target application logics [1]. We are particularly working on two urban computing projects regarding the modelling and management of crowds and smart energy management in urban clusters.…”

“…These pipelines provide analytics visions of data to target applications. H-STREAM relies on (i) message queues for collecting streams online from IoT farms; and (ii) a backend execution environment [1] that provides a virtual data centre with infrastructure resources necessary for executing costly processes.…”

Graph Analytics Workflows Enactment on Just in Time Data Centres Position Paper

“…So a virtual entity that dynamically provides resources that touch the edge, fog and data centre according to the workloads submitted by the workflows and their Service Level Objectives (SLOs). Therefore, this paper discusses our vision about multirolecapable decision-making systems across a broad range of DS workflows working on graphs through an agile, autonomous, composable, and resilient "Just-in-Time Architecture" for DS Pipelines (JITA-4DS) [1].…”