Sampling operations on big data

Gadepally, Vijay; Herr, Taylor; Johnson, Luke B.; Milechin, Lauren; Milosavljevic, Maja; Miller, Benjamin A.

doi:10.1109/acssc.2015.7421398

Cited by 10 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, two problems (memory/time) are based on sampling in the Big Data context. As evidence, we cite, just as an example but not limited to, some related works, such as the one done by Gadepally et al (2015) in which they treated predictive analysis so that they exhibit the effect of sampling in social network links. There is also another work no less important than the others presented by Albattah (2016).…”

Section: Related Workmentioning

confidence: 99%

Big Data analytics for prediction: parallel processing of the big learning base with the possibility of improving the final result of the prediction

Djafri

Bensaber

Adjoudj

2018

IDD

View full text Add to dashboard Cite

Purpose This paper aims to solve the problems of big data analytics for prediction including volume, veracity and velocity by improving the prediction result to an acceptable level and in the shortest possible time. Design/methodology/approach This paper is divided into two parts. The first one is to improve the result of the prediction. In this part, two ideas are proposed: the double pruning enhanced random forest algorithm and extracting a shared learning base from the stratified random sampling method to obtain a representative learning base of all original data. The second part proposes to design a distributed architecture supported by new technologies solutions, which in turn works in a coherent and efficient way with the sampling strategy under the supervision of the Map-Reduce algorithm. Findings The representative learning base obtained by the integration of two learning bases, the partial base and the shared base, presents an excellent representation of the original data set and gives very good results of the Big Data predictive analytics. Furthermore, these results were supported by the improved random forests supervised learning method, which played a key role in this context. Originality/value All companies are concerned, especially those with large amounts of information and want to screen them to improve their knowledge for the customer and optimize their campaigns.

show abstract

Section: Related Workmentioning

confidence: 99%

Big Data analytics for prediction: parallel processing of the big learning base with the possibility of improving the final result of the prediction

Djafri

Bensaber

Adjoudj

2018

IDD

View full text Add to dashboard Cite

show abstract

“…For future work, we are interested in developing a greater suite of algorithms that can be directly developed using Graphulo such as dimensional analysis [20] and big data sampling [21]. Further, we would like to investigate providing a Pig interface to Graphulo as well as calling Graphulo operations from the Apache Spark framework.…”

Section: Discussionmentioning

confidence: 99%

Benchmarking the graphulo processing framework

Weale

Gadepally

Hutchison

et al. 2016

2016 IEEE High Performance Extreme Computing Conference (HPEC)

Self Cite

View full text Add to dashboard Cite

Abstract-Graph algorithms have wide applicablity to a variety of domains and are often used on massive datasets. Recent standardization efforts such as the GraphBLAS specify a set of key computational kernels that hardware and software developers can adhere to. Graphulo is a processing framework that enables GraphBLAS kernels in the Apache Accumulo database. In our previous work, we have demonstrated a core Graphulo operation called TableMult that performs large-scale multiplication operations of database tables. In this article, we present the results of scaling the Graphulo engine to larger problems and scalablity when a greater number of resources is used. Specifically, we present two experiments that demonstrate Graphulo scaling performance is linear with the number of available resources. The first experiment demonstrates cluster processing rates through Graphulo's TableMult operator on two large graphs, scaled between 2 17 and 2 19 vertices. The second experiment uses TableMult to extract a random set of rows from a large graph (2 19 nodes) to simulate a cued graph analytic. These benchmarking results are of relevance to Graphulo users who wish to apply Graphulo to their graph problems.

show abstract

“…Data preparation generates representative Big Data samples that serve as an entry for profiling, quality evaluation, and quality rules validation. (a) Sampling: Several sampling strategies can be applied to Big Data as surveyed in [54,55]. In this work, the authors evaluated the effect of sampling methods on Big Data and concluded that the sampling of large datasets reduces the run-time and computational footprint of link prediction algorithms, maintaining an adequate prediction performance.…”

Section: Data Preparation: Sampling and Profilingmentioning

confidence: 99%

Big data quality framework: a holistic approach to continuous quality management

et al. 2021

View full text Add to dashboard Cite

Big Data is an essential research area for governments, institutions, and private agencies to support their analytics decisions. Big Data refers to all about data, how it is collected, processed, and analyzed to generate value-added data-driven insights and decisions. Degradation in Data Quality may result in unpredictable consequences. In this case, confidence and worthiness in the data and its source are lost. In the Big Data context, data characteristics, such as volume, multi-heterogeneous data sources, and fast data generation, increase the risk of quality degradation and require efficient mechanisms to check data worthiness. However, ensuring Big Data Quality (BDQ) is a very costly and time-consuming process, since excessive computing resources are required. Maintaining Quality through the Big Data lifecycle requires quality profiling and verification before its processing decision. A BDQ Management Framework for enhancing the pre-processing activities while strengthening data control is proposed. The proposed framework uses a new concept called Big Data Quality Profile. This concept captures quality outline, requirements, attributes, dimensions, scores, and rules. Using Big Data profiling and sampling components of the framework, a faster and efficient data quality estimation is initiated before and after an intermediate pre-processing phase. The exploratory profiling component of the framework plays an initial role in quality profiling; it uses a set of predefined quality metrics to evaluate important data quality dimensions. It generates quality rules by applying various pre-processing activities and their related functions. These rules mainly aim at the Data Quality Profile and result in quality scores for the selected quality attributes. The framework implementation and dataflow management across various quality management processes have been discussed, further some ongoing work on framework evaluation and deployment to support quality evaluation decisions conclude the paper.

show abstract

Sampling operations on big data

Cited by 10 publications

References 13 publications

Big Data analytics for prediction: parallel processing of the big learning base with the possibility of improving the final result of the prediction

Big Data analytics for prediction: parallel processing of the big learning base with the possibility of improving the final result of the prediction

Benchmarking the graphulo processing framework

Big data quality framework: a holistic approach to continuous quality management

Contact Info

Product

Resources

About