Dynamic Control of Data Streaming and Processing in a Virtualized Environment

Abstract

Abstract-Performance of data streaming applications is co-determined by both networking and computing resources, and therefore they should be co-scheduled and co-allocated in an integrated and coordinated way. Dynamic control of resource scheduling and allocation is required, because unilateral redundancy in either networking or computing resources may result in the overprovision of it and the other may become a bottleneck. To avoid resource shortage as well as overprovision, in this paper, a virtualized platform is utilized to implement data streaming and processing. In this platform, fuzzy logic controllers are designed to allocate CPU resources; iterative bandwidth allocation is applied and is processing-and storage-aware to guarantee on-demand data provisioning. Experimental results show that our approach leads to higher application performance as well as higher resource utilization, compared with other resource scheduling and allocation methods.Note to Practitioners-Data streaming has become an important paradigm in many business and scientific applications, such as fraud detection in banking industry and gravitational-wave observation in astronomy. In this paper, we apply virtualization to provide better support for these applications, leveraging its advantage in resource on-demand allocation. A novel dynamic control method is proposed so that CPU and bandwidth can be co-scheduled and co-allocated since for data streaming applications these resources are tightly coupled from the performance perspective. In this control method, fuzzy logic control is applied for CPU allocation and an iterative algorithm is adopted for processing-, congestion-, and storage-aware bandwidth allocation. We use fuzzy control because it does not rely on the mathematical modeling of an object and can implement human experts' heuristic knowledge via IF-THEN rules. Therefore, it is desirable in our data streaming scenario, due to the variable coupling and heavily nonlinear nature of the system.

Enabling Distributed Computing Systems with ElopTM

“…Internet of Things (IoT) 8 . The element data can be extended to an element sensor, supporting continuous data streaming [10]. In general, sensors play embedded between cyber and physical spaces, enabling distributed computing for IoT applications [14].…”

“…To solve the A4 problem, the physical properties of computing elements are totally abstracted and described in a logical way. Resource management and scheduling can be also fulfilled at the logical layer [10]. In addition, dynamic construction of virtual organizations (VO) is supported for security, trust and privacy management of various computing elements [8].…”

Performance modeling and analysis of virtualized multi-tier applications under dynamic workloads

“…Furthermore, two RUBiS instances were benchmarked in dynamic workload condition to evaluate the proposed model. Several attempts have been made to use rule-based modeling such as fuzzy logic controlling to provide subtle resource provisioning of multi-tier applications in Cloud environment (Lama, 2012;Cao et al, 2012). However, accurate performance modeling of complex nature of multi-tier Web applications based on such approaches is a complex task.…”

Research and Applications of Cloud Manufacturing in China

“…Besides, processing method for big data of manufacturing services based on Relational Database and Hadoop framework has proposed. And virtual resources optimization scheduling, fault-tolerance and monitoring technologies have been proposed (Cao et al 2012;Bao et al 2012;Zhang et al 2013). …”

“…Fuzzy logic controllers are designed to allocate CPU resources. Besides, iterative bandwidth allocation is applied and is processing and storageaware to guarantee on-demand data provisioning (Cao et al 2012).…”