Supporting Streaming Updates in an Active Data Warehouse

Polyzotis, Neoklis; Skiadopoulos, Spiros; Vassiliadis, Panos; Simitsis, Alkis; Frantzell, N. E.

doi:10.1109/icde.2007.367893

Cited by 77 publications

(63 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We do not consider the ordering of stream tuples in the join output as it is not important particularly in the scenario of data warehousing. To evaluate the approach, we combine the front-stage with two known semi-stream algorithms: (i) MESHJOIN [10,11], (ii) HYBRIDJOIN [12]. Our experimental results demonstrate the performance benefit of the approach.…”

mentioning

confidence: 97%

“…A characteristic of MESHJOIN is that it performs a staggered execution of the hash table build in order to load in stream tuples more steadily. The algorithm makes no assumptions about data distribution and the organization of the master data, and the MESHJOIN authors report that the algorithm performs worse with skewed data [10,11]. We previously presented a variant, R-MESHJOIN (reduced Mesh Join) [23], in order to clarify certain dependencies among the components of MESHJOIN.…”

Section: Related Workmentioning

confidence: 99%

“…MESHJOIN (Mesh Join) [10,11] has been designed especially for joining a continuous stream with a disk-based relation, like the scenario in active data warehouses. MESHJOIN is a hash join where the stream serves as the build input and the disk-based relation serves as the probe input.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Skewed distributions in semi-stream joins: How much can caching help?

Naeem

Dobbie

Lutteroth

et al. 2017

Information Systems

View full text Add to dashboard Cite

Semi-stream join algorithms join a fast data stream with a disk-based relation. This is important, for example, in real-time data warehousing where a stream of transactions is joined with master data before loading it into a data warehouse. In many important scenarios, the stream input has a skewed distribution, which makes certain performance optimisations possible.We propose two such optimisation techniques: 1) a caching technique for frequently used master data, and 2) a technique for selective load shedding of stream tuples. The caching technique is fine-grained, operating on a tuple-level. Furthermore, it is generic in the sense that it can be applied to different semistream join algorithms to deal with high loads. We analyze it by combining it with various well-known semi-stream joins, and show that it improves the service rate by more than 40% for typical data with skewed distributions. The load shedding technique sheds the fraction of the stream that is most expensive to join. In contrast to existing approaches, the service rate improves under load shedding. We present experimental data showing significant improvements as compared to related approaches and perform a sensitivity analysis for various internal parameters.

show abstract

mentioning

confidence: 97%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Skewed distributions in semi-stream joins: How much can caching help?

Naeem

Dobbie

Lutteroth

et al. 2017

Information Systems

View full text Add to dashboard Cite

show abstract

“…The common case for updates to a data warehouse is the Extract-TransformLoad (ETL) processing [25], i.e., data are extracted from the sources and loaded to the data warehouse during specified time intervals. As applications are pushing for higher levels of freshness, data warehouses are updated as frequently as possible, giving rise to Active Data Warehousing [13,22,21].…”

Section: Previous Workmentioning

confidence: 99%

Strategic Management for Real-Time Business Intelligence

Zoumpatianos

Palpanas

Mylopoulos

2013

Lecture Notes in Business Information Processing

View full text Add to dashboard Cite

Abstract. Even though much research has been devoted on real-time data warehousing, most of it ignores business concerns that underlie all uses of such data. The complete Business Intelligence (BI) problem begins with modeling and analysis of business objectives and specifications, followed by a systematic derivation of real-time BI queries on warehouse data. In this position paper, we motivate the need for the development of a complete Real Time BI stack able to continuously evaluate and reason about strategic objectives. We argue that an integrated system, able to receive formal specifications of the organization's strategic objectives and to transform them into a set of queries that are continuously evaluated against the warehouse, offers significant benefits. In this context, we propose the development of a set of real-time query answering mechanisms able to identify warehouse segments with temporal patterns of special interest, as well as novel techniques for mining warehouse regions that represent expected, or unexpected threats and opportunities. With such a vision in mind, we propose an architecture for such a framework, and discuss relevant challenges and research directions.

show abstract

“…A novel stream-based equijoin algorithm, MESHJOIN (N. Polyzotis, Skiadopoulos, Vassiliadis, Simitsis, & Frantzell, 2007) (Neoklis Polyzotis, Skiadopoulos, Vassiliadis, Simitsis, & Frantzell, 2008) is in principle a hash join, where the stream serves as the build input and the diskbased relation serves as the probe input. The main contribution is a staggered execution of the hash table build and an optimization of the disk buffer for the disk-based relation.…”

Section: Figure 1: An Example Of Stream-based Joinmentioning

confidence: 99%