Big Data Analytics Has Little to Do with Analytics

Abstract

As big data analytics is adapted across multitude of domains and applications there is a need for new platforms and architectures that support analytic solution engineering as a lean and iterative process. In this paper we discuss how different software development processes can be adapted to data analytic process engineering, incorporating service oriented architecture, scientific workflows, model driven engineering and semantic technology. Based on the experience obtained through ADAGE framework [1] and the findings of the survey on how semantic modeling is used for data analytic solution engineering [6], we propose two research directions -big data analytic development lifecycle and data analytic knowledge management for lean and flexible data analytic platforms.

Semantic modeling for engineering data analytics solutions

“…Recommended future work, discussed in section 5.2, emphasizes the importance of moving semantic technology out of certain research silos and aims at developing new research agendas around capturing highlevel intents and goals of data analysts and translating them to executable analytics processes, incorporating a multitude of well-defined semantic knowledge repositories that have the capacity to be developed, expanded and maintained independently from each other. This can be achieved within established software engineering frameworks, but they need to be specifically tai-lored to the particular characteristics of the DAS engineering life-cycle as presented in [6]. As the next stage of this research effort, authors are working on designing a requirement driven platform that provides support for end-to-end analytics process engineering, incorporating semantic concept types identified through this mapping study [51,56].…”

“…The ontologies should incorporate knowledge related to domain concepts and business goals as well as the concepts useful for the execution level. For example, an ontological representation of a data source may contain information necessary to retrieve data, but also information about the data quality, the latency of data acquisition, metadata that can be used to decide which algorithm is suitable to process the data (e.g.the 6 www.obofoundry.org/ knowledge of whether the data is time-series or not can be used to reduce the set of algorithms should consider) and the relationship between the data and other concepts. Representation of existing knowledge and enabling efficient reuse of accumulated knowledge and resources can reduce the effort spent on expert consultations or employee training.…”

“…A DAS can be itself decomposed into multi-ple tasks such as identifying suitable datasets, developing and validating analytics models and interpreting final results. The software engineering aspect of DAS, which we refer to as "DAS engineering", involves designing, developing and deploying DASs and includes tasks such as requirement elicitation, data integration and process composition [6].…”

Software-driven big data analytics

“…On one hand, big data itself can cause significant performance problems in application programs in general, especially when involving databases [15]. On the other hand, following the No-Free-Lunch theorem [24], various data types and analytical demands might require completely different BDA applications involving different time and space complexities [6]. For example, de facto BDA workload characteristics tremendously vary, and the typical ones include batchprocessing for offline analytical jobs, streamprocessing for real-time processing of data, query-processing with transactional features, and even a combination of them [16].…”

“…Given the aforementioned software nature of BDA applications, software engineering can act as a key to addressing the existing challenges in the BDA domain and to supporting different areas and aspects of BDA practices. It is even claimed that BDA has little to do with analytics but with software engineering [24]. From the software developer's perspective, the theories, processes, and techniques of software engineering can be introduced to the realization of efficient analytical operations [22].…”