T-Storm: Traffic-Aware Online Scheduling in Storm

Xu, Jiaojiao; Chen, Zhenhua; Tang, Jian; Su, Sen

doi:10.1109/icdcs.2014.61

Cited by 187 publications

(136 citation statements)

References 16 publications

Supporting

Mentioning

133

Contrasting

Unclassified

Order By: Relevance

“…T-Storm [91] (i.e. Traffic-aware Storm), for instance, aims to reduce inter-process and inter-node communication, which is shown to degrade performance under certain workloads.…”

Section: Online Techniquesmentioning

confidence: 99%

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

Assuno

Veith

Buyya

2018

Journal of Network and Computer Applications

257

View full text Add to dashboard Cite

Under several emerging application scenarios, such as in smart cities, operational monitoring of large infrastructure, wearable assistance, and Internet of Things, continuous data streams must be processed under very short delays. Several solutions, including multiple software engines, have been developed for processing unbounded data streams in a scalable and efficient manner. More recently, architecture has been proposed to use edge computing for data stream processing. This paper surveys state of the art on stream processing engines and mechanisms for exploiting resource elasticity features of cloud computing in stream processing. Resource elasticity allows for an application or service to scale out/in according to fluctuating demands. Although such features have been extensively investigated for enterprise applications, stream processing poses challenges on achieving elastic systems that can make efficient resource management decisions based on current load. Elasticity becomes even more challenging in highly distributed environments comprising edge and cloud computing resources. This work examines some of these challenges and discusses solutions proposed in the literature to address them.

show abstract

“…T-Storm [91] (i.e. Traffic-aware Storm), for instance, aims to reduce inter-process and inter-node communication, which is shown to degrade performance under certain workloads.…”

Section: Online Techniquesmentioning

confidence: 99%

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

Assuno

Veith

Buyya

2018

Journal of Network and Computer Applications

257

View full text Add to dashboard Cite

show abstract

“…Besides the elasticity extension to System S, there are also several proposed extensions to Apache Storm, which replace the default scheduler with custom implementations to optimize the parallelization of operators as well as the placement thereof on different computational resources. Two of these approaches have been presented by Aniello, Baldoni & Querzoni (2013) and Xu et al (2014). These two publications present threshold-based custom schedulers, which can adopt the topology deployment at runtime, depending on the incoming data volume and the actual load for Apache Storm.…”

Section: Related Workmentioning

confidence: 99%

Cost-efficient enactment of stream processing topologies

Hochreiner

Vögler

Schulte

et al. 2017

PeerJ Computer Science

View full text Add to dashboard Cite

The continuous increase of unbound streaming data poses several challenges to established data stream processing engines. One of the most important challenges is the cost-efficient enactment of stream processing topologies under changing data volume. These data volume pose different loads to stream processing systems whose resource provisioning needs to be continuously updated at runtime. First approaches already allow for resource provisioning on the level of virtual machines (VMs), but this only allows for coarse resource provisioning strategies. Based on current advances and benefits for containerized software systems, we have designed a cost-efficient resource provisioning approach and integrated it into the runtime of the Vienna ecosystem for elastic stream processing. Our resource provisioning approach aims to maximize the resource usage for VMs obtained from cloud providers. This strategy only releases processing capabilities at the end of the VMs minimal leasing duration instead of releasing them eagerly as soon as possible as it is the case for threshold-based approaches. This strategy allows us to improve the service level agreement compliance by up to 25% and a reduction for the operational cost of up to 36%. Subjects

show abstract

“…It includes spouts and bolts, which could be executed with many tasks in parallel on multiple machines (worker nodes) in a cluster [8]. Storm is designed to process unbounded streams of data in a storm cluster (master node and worker nodes).…”

Section: Related Workmentioning

confidence: 99%

“…Storm is designed to process unbounded streams of data in a storm cluster (master node and worker nodes). However, as discussed by Xu et al [8], Storm weakly focuses on the performance and job assignment with different workload. Moreover, the deployment of Storm requires more nodes.…”

Section: Related Workmentioning

confidence: 99%

“…It is a distributed, reliable, and fault-tolerant system, which is used particularly for processing continuous real-time data streams. It adopts a default scheduler, which employs a simple round-robin algorithm based cluster, without considering the internode and interprocess traffic, and it may have a significant impact on performance [8]. Besides this, the default scheduler in Storm often uses all available nodes in a cluster, regardless of workload.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Novel Complex Event Processing Engine for Intelligent Data Analysis in Integrated Information Systems

Wang

Zhou

Ali

et al. 2016

International Journal of Distributed Sensor Networks

View full text Add to dashboard Cite

Novel and effective engines for data analysis in integrated information systems are urgently required by diverse applications, in which massive business data can be analyzed to enable the capturing of various situations in real time. The performance of existing engines has limited capacity of data processing in distributed computing. Although Complex Event Processing (CEP) has enhanced the capacity of data analysis in information systems, it is still a big challenging task since events are rapidly increasing in diverse applications. In this paper, a lightweight intelligent data analysis system with a novel CEP engine named LIDA-E is introduced, which employs the knowledge base with rules and an event processing algorithm for analysis. Event models as well as operators support the rules for event selection and aggregation. These operators and rules have been utilized for constructing new CEP system architecture which combines expressiveness and efficiency in analysis. It adopts the agents and filter conception explicitly to provide the event transmission mechanism efficiently. Finally, the comparison between the proposed engine and the existing engine shows that LIDA-E has 48.65% averagely reduced time cost in different tests. The experimental results demonstrate that the developed architecture has better performance in both transmitting and analyzing a large number of events.

show abstract

T-Storm: Traffic-Aware Online Scheduling in Storm

Cited by 187 publications

References 16 publications

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

Distributed data stream processing and edge computing: A survey on resource elasticity and future directions

Cost-efficient enactment of stream processing topologies

A Novel Complex Event Processing Engine for Intelligent Data Analysis in Integrated Information Systems

Contact Info

Product

Resources

About