Data integration flows for business intelligence

A Unified View of Data-Intensive Flows in Business Intelligence Systems: A Survey

“…To this end, in this paper, we have given a unified view of data-intensive flows, focusing on the challenges that next generation BI setting has brought. Currently, even though many works under different names (i.e., from different perspectives) have envisioned and/or proposed conceptual frameworks for next generation BI ecosystems (e.g., [1,6,15,17,22,63]), we still lack an end-to-end solution for managing the complete lifecycle of data-intensive flows. Going back to Tables 1, 2, and 3, we can observe a certain overlapping of levels of different dimensions between the theoretical problem of data exchange and data warehousing approaches (i.e., ETL), as well as between data integration and data-intensive flows in the next generation BI setting (i.e., ETO).…”

“…In the case of partially materializing data, as suggested in [15], the Flow Designer module should be aware or be able to reconstruct the target schema, where data are loaded in a previously executed batch process, and further queried when interactively answering information requirements. Thus, the final data flow ready for deployment must be integrated from the combination of data flows that directly access data sources and querying previously materialized target data (i.e., the Flow Integrator module).…”

“…In Phase I, we focused on the keywords for finding seminal works in the DW and ETL field (i.e., "data warehousing", "ETL", "business intelligence"), as well as the most relevant works on the next generation BI (i.e., "next generation business intelligence", "BI 2.0", "data-intensive flows", "operational business intelligence"). While in the case of traditional DW and ETL approaches we encountered and targeted the most influential books from the field (e.g., [53,46]) and some extensive surveys (e.g., [96]), in the case of the next generation BI approaches, we mostly selected surveys or visionary papers on the topic (e.g., [1,2,17,15,40]). Furthermore, the following phases included keyword search based on the terminology found in the created study outline (see Figure 4), as well as the identified dimensions (see Figure 5).…”

“…The next generation BI setting goes by various names: operational BI (e.g., [15,22]), live BI (e.g., [17]), collaborative BI (e.g., [6]), self-service BI (e.g., [1]), situational BI (e.g., [63]). While these works look at the problem from different perspectives, in general, they all aim at enabling the broader spectrum of business users to access a plethora of heterogeneous sources (not all being under the control of the user's organization and known in advance), and to extract, transform and combine these data, in order to make right-time decisions.…”

“…The "right" (Medium) level of interactivity is also discussed to be crucial in the next generation BI setting (e.g., [15]) where a partial materialization is envisioned for a subset of data with low latency and low freshness requirements (i.e., for which we can rely on the last batched ETL run). Following the similar idea, [74] proposes a framework for combining data-intensive flows with user query pipelines and hence choosing the optimal materialziation point (i.e., Medium interactivity) in the data flow, based on different cost metrics (e.g., source update rates and view maintenance costs).…”