Modern Stream Processing Engines (SPEs) process large data volumes under tight latency constraints. Many SPEs execute processing pipelines using message passing on shared-nothing architectures and apply a partition-based scale-out strategy to handle high-velocity input streams. Furthermore, many state-of-the-art SPEs rely on a Java Virtual Machine to achieve platform independence and speed up system development by abstracting from the underlying hardware. In this paper, we show that taking the underlying hardware into account is essential to exploit modern hardware efficiently. To this end, we conduct an extensive experimental analysis of current SPEs and SPE design alternatives optimized for modern hardware. Our analysis highlights potential bottlenecks and reveals that state-of-the-art SPEs are not capable of fully exploiting current and emerging hardware trends, such as multi-core processors and high-speed networks. Based on our analysis, we describe a set of design changes to the common architecture of SPEs to scale-up on modern hardware. We show that the single-node throughput can be increased by up to two orders of magnitude compared to state-of-the-art SPEs by applying specialized code generation, fusing operators, batch-style parallelization strategies, and optimized windowing. This speedup allows for deploying typical streaming applications on a single or a few nodes instead of large clusters.

No abstract

No abstract

The optimization of the water resource usage in hydrothermal electric energy systems is crucial to assure an economic and reliable load supply. The long term hydrothermal scheduling is a complex problem mainly due to the randomness of the inflows and the nonlinearity of hydro production and thermal cost functions. Some optimization approaches have been proposed, including the Stochastic Dynamic Programming (SDP), which is one of the most commonly used techniques to deal with this problem. Its computational requirements, however, tend to be heavy and, as a result, its application on real systems is limited. In this paper we proposed the use of an Adaptative Neuro-Fuzzy Inference System in parallel with a deterministic optimization model as a simpler and less complex alternative approach to the hydrothermal scheduling. The information of the optimal operation is processed by the network that produces fuzzy rules describing the optimal decisions. The performance of the proposed approach was compared to other policies, including SDP, by simulation using historical inflows records of Emborcação, a large Brazilian hydroelectric power plant. Results demonstrated that the Neuro-Fuzzy approach provided similar performance to the more computationally complex and commonly used SDP.

No abstract