Addressing big data variety using an automated approach for data characterization

Abstract

The creation of new knowledge from manipulating and analysing existing knowledge is one of the primary objectives of any cognitive system. Most of the effort on Big Data research has been focussed upon Volume and Velocity, while Variety, “the ugly duckling” of Big Data, is often neglected and difficult to solve. A principal challenge with Variety is being able to understand and comprehend the data. This paper proposes and evaluates an automated approach for metadata identification and enrichment in describing Big Data. The paper focuses on the use of self-learning systems that will enable automatic compliance of data against regulatory requirements along with the capability of generating valuable and readily usable metadata towards data classification. Two experiments towards data confidentiality and data identification were conducted in evaluating the feasibility of the approach. The focus of the experiments was to confirm that repetitive manual tasks can be automated, thus reducing the focus of a Data Scientist on data identification and thereby providing more focus towards the extraction and analysis of the data itself. The origin of the datasets used were Private/Business and Public/Governmental and exhibited diverse characteristics in relation to the number of files and size of the files. The experimental work confirmed that: (a) the use of algorithmic techniques attributed to the substantial decrease in false positives regarding the identification of confidential information; (b) evidence that the use of a fraction of a data set along with statistical analysis and supervised learning is sufficient in identifying the structure of information within it. With this approach, the issues of understanding the nature of data can be mitigated, enabling a greater focus on meaningful interpretation of the heterogeneous data.

LSDStrategy: A Lightweight Software-Driven Strategy for Addressing Big Data Variety of Multimedia Streaming

“…These three approaches relied on the semantic parsing of the workload in-depth as a limited approach [22]. Moreover, a softwaredriven model-based non-semantic parsing for the big data workload was proposed in [31]. The proposed method based on metadata identification in big data focuses on the application of self-learning systems to enable automatic data compliance with legal requirements and the possibility of providing essential and easily accessible metadata for data classification.…”

Intuitionistic and Interval-Valued Fuzzy Set Representations for Data Mining

“…Therefore, if there is no uncertainty in the membership function, this reduces to ordinary fuzzy sets. TY2(d) = {< (a i , r), m T (a i ,r) > | a i ∈ D} (7) where r ∈ P x ⊆ [0, 1].…”

“…This can include sensor information and data ranging to the subjective interpretations obtained from expert individuals and analysts. Currently, increasingly massive amounts of heterogeneous data and information from multiple sources are prevalent where the problems of Big Data are being managed [4][5][6][7]. However, although effective decision making should be able to make use of all the available and relevant information about such combined uncertainty, an assessment of the value of a generalization result is critical.…”

Data‐driven predictions of summertime visits to lakes across 17 US states

“…Another analytical difficulty faced by analysts when using “big data” is that, unlike more consciously constructed research projects where data collection and construction can be carefully planned, datasets collected via mobile computing, data digitization, and social media are often an amalgam of various data streams that, when cobbled together, have missing values for many observations (Einav & Levin, 2014; Emmanuel et al, 2021; Fan et al, 2014; Vranopoulos et al, 2022). When this occurs, a modeler may be tempted to ignore every observation that does not have a complete set of predictive attributes.…”