Bleach: A Distributed Stream Data Cleaning System

Tian, Yongchao; Michiardi, Pietro; Vukolić, Marko

doi:10.1109/bigdatacongress.2017.24

Cited by 9 publications

(8 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, their work assumes a fixed ontology and do not provide a real application scenario. Furthermore, the Bleach system [17] focusses on detection and cleaning of inconsistent data. The rule set is adaptable as in our case.…”

Section: Discussion and Related Workmentioning

confidence: 99%

“…This set can be processed in parallel, since the products are independent of each other (R9). Windows are needed on a stream, since the amount of data would otherwise increase infinitely [17].…”

Section: No Additional Messages Additional Messages Can Be Anmentioning

confidence: 99%

“…Furthermore, the analyses are performed based on potentially erroneous data. In [17] and [16] the authors pointed out that high quality data is a decisive factor for meaningful analyses. In our scenario, data quality refers to inconsistencies within the transmitted information from the machines.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Enhancing data quality and process optimization for smart manufacturing lines in industry 4.0 scenarios

Paasche

Groppe

2022

Proceedings of the International Workshop on Big Data in Emergent Distributed Environments

View full text Add to dashboard Cite

An essential component of today's industry is data, which is generated during manufacturing. The goal of industry 4.0 is efficient collection, processing and analysis of this data. In our work, we address these three tasks and present an extensible system to solve them. To the best of our knowledge, the combination of a consistency checker (CC) for data preparation and a digital twin (DT) for analysis activities represents a novel approach. Consistency checking in combination with a DT leads to increased data quality, which in turn has a positive effect on analyses, like reducing errors to decrease costs, identifying relevant parameters to increase the productivity, and determining the bottleneck of a manufacturing line for enhanced production planning.

show abstract

Section: Discussion and Related Workmentioning

confidence: 99%

Section: No Additional Messages Additional Messages Can Be Anmentioning

confidence: 99%

See 1 more Smart Citation

Enhancing data quality and process optimization for smart manufacturing lines in industry 4.0 scenarios

Paasche

Groppe

2022

Proceedings of the International Workshop on Big Data in Emergent Distributed Environments

View full text Add to dashboard Cite

show abstract

“…The solution proposed for detecting and repairing dirty data in Gohel et al (2017) resolves errors like inconsistency, lack of accuracy, and redundancy, by treating multiple types of quality rules holistically. In Tian et al (2017), a rule-based data-cleaning technique is proposed where a set of rules defines how data should be cleaned. Moreover, the research of Zhang et al (2017) shows an innovative method for correcting values in time series that are considered abnormal, through anomaly detection, where the authors are using the method of iterative minimum repairing (IMR).…”

Section: Policycloud Ingest Analyticsmentioning

confidence: 99%

PolicyCLOUD: A prototype of a cloud serverless ecosystem for policy analytics

et al. 2022

View full text Add to dashboard Cite

We present PolicyCLOUD: a prototype for an extensible serverless cloud-based system that supports evidence-based elaboration and analysis of policies. PolicyCLOUD allows flexible exploitation and management of policy-relevant dataflows, by enabling the practitioner to register datasets and specify a sequence of transformations and/or information extraction through registered ingest functions. Once a possibly transformed dataset has been ingested, additional insights can be retrieved by further applying registered analytic functions to it. PolicyCLOUD was built as an extensible framework toward the creation of an analytic ecosystem. As of now, we have developed several essential ingest and analytic functions that are built-in within the framework. They include data cleaning, enhanced interoperability, and sentiment analysis generic functions; in addition, a trend analysis function is being created as a new built-in function. PolicyCLOUD has also the ability to tap on the analytic capabilities of external tools; we demonstrate this with a social dynamics tool implemented in conjunction with PolicyCLOUD, and describe how this stand-alone tool can be integrated with the PolicyCLOUD platform to enrich it with policy modeling, design and simulation capabilities. Furthermore, PolicyCLOUD is supported by a tailor-made legal and ethical framework derived from privacy/data protection best practices and existing standards at the EU level, which regulates the usage and dissemination of datasets and analytic functions throughout its policy-relevant dataflows. The article describes and evaluates the application of PolicyCLOUD to four families of pilots that cover a wide range of policy scenarios.

show abstract

“…Other algorithms in the literature exploit the violation graph approach, which is based on user-defined rules, to perform data cleaning, such as the equivalence class algorithm [22] or the holistic data cleaning algorithm [23]. In [24], a stream data cleaning system for categorical and numerical data is proposed, which relies on compact data structures to maintain the necessary state to repair data. Dirty data are repaired using the concept of distributed violation graph, which is an extension of the violation graph approach, aimed to improve the scalability performances.…”

Section: Related Workmentioning

confidence: 99%