Data Cleaning-A Thorough Analysis and Survey on Unstructured Data

Kumar, Vinay; Khosla, Cherry

doi:10.1109/confluence.2018.8442950

Cited by 30 publications

(5 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors of [52] analyzed and visualized unstructured data on air pollution, demonstrating the prevalence of numerous dangerous gases in the atmosphere. The authors emphasized the need for data cleaning in dealing with the problems caused by dirty data, such as null values, incomplete data, missing values, and inconsistent sampling date formats.…”

Section: Cleaning Datamentioning

confidence: 99%

A Model for Enhancing Unstructured Big Data Warehouse Execution Time

Farhan,

Youssef,

Abdelhamid

2024

BDCC

View full text Add to dashboard Cite

Traditional data warehouses (DWs) have played a key role in business intelligence and decision support systems. However, the rapid growth of the data generated by the current applications requires new data warehousing systems. In big data, it is important to adapt the existing warehouse systems to overcome new issues and limitations. The main drawbacks of traditional Extract–Transform–Load (ETL) are that a huge amount of data cannot be processed over ETL and that the execution time is very high when the data are unstructured. This paper focuses on a new model consisting of four layers: Extract–Clean–Load–Transform (ECLT), designed for processing unstructured big data, with specific emphasis on text. The model aims to reduce execution time through experimental procedures. ECLT is applied and tested using Spark, which is a framework employed in Python. Finally, this paper compares the execution time of ECLT with different models by applying two datasets. Experimental results showed that for a data size of 1 TB, the execution time of ECLT is 41.8 s. When the data size increases to 1 million articles, the execution time is 119.6 s. These findings demonstrate that ECLT outperforms ETL, ELT, DELT, ELTL, and ELTA in terms of execution time.

show abstract

Section: Cleaning Datamentioning

confidence: 99%

A Model for Enhancing Unstructured Big Data Warehouse Execution Time

Farhan,

Youssef,

Abdelhamid

2024

BDCC

View full text Add to dashboard Cite

show abstract

“…Data standardization can pave the way for the use of data science and machine learning driven innovations in the battery development industry. The standardization of conventions and definitions for data are the first step to any data science or machine learning project, where "data cleaning" as it is referred to colloquially amongst data scientists, is known to comprise a substantial portion of the effort in any large-scale data science project (Chu et al, 2016;Kumar and Khosla, 2018;Petrova-Antonova and Tancheva, 2020;Wang and Wang, 2020;Ilyas and Rekatsinas, 2022).…”

Section: Data Science Machine Learning and Battery Datamentioning

confidence: 99%

Voltaiq data format—A standard data format for collection of battery data to enable big data comparisons and analyses across the battery lifecycle

et al. 2022

View full text Add to dashboard Cite

Batteries have enabled modernization of society through portability of electricity. Batteries are also a crucial component to enabling clean technologies of the future such as grid storage and electrified transportation. Because of their ubiquity in modern society, global organizations develop and commercialize batteries for their electrified products. Across the field of battery development, in both commercial and academic settings, there is broad utility in standardization of data formats amongst disparate data sources, labs, equipment, organizations, industries, and lifecycle phases. Due to the way the nascent industry developed, there is a lack of standardization for how performance data is recorded, which is now hindering the industry’s ability to learn from data and accelerate growth. Herein, we describe the different types of data, formats, conventions, and standardization for each phase in the battery lifecycle. Next, we provide a standard data format and conventions for the community to either utilize in their data collection practices or map their existing data into: the Voltaiq Data Format (VDF). This standard data format provides the flexibility needed to capture the variability in data formats and conventions along the battery lifecycle. The utility of this standard format aids in collaboration within and across organizations, accelerating innovation across the industry, and paves the way for the battery community to start utilizing the power of machine learning and data science.

show abstract

“…The promising technologies essential for the organization of the digital industry in businesses and the collection of technologies needed to ensure the transformation from the current state of the industry to Industry 4.0 and then to Industry 5.0. A formal overview of Industry 4.0 and Industry 5.0 is also given, allowing the problem to be presented as a mathematical problem [7].…”

Section: Literature Reviewmentioning

confidence: 99%

“…Hasan et al developed natural language processing (NLP) based preprocessed data framework to evaluate sentiment, and integrated the model definition of Bag of Words (BoW) and Term Frequency-Inverse Text Frequency (TF-IDF) [7].…”

Section: Literature Reviewmentioning

confidence: 99%

SoloDB for Social Media’s Big Data Using Deep Natural Language with AI Applications and Industry 5.0

Devi

Selvam

2021

Smart Innovation, Systems and Technologies

View full text Add to dashboard Cite

Deep natural language processing is an algorithmic approach that enables computers to understand language using patterns, purpose, adequate experience, and a natural human data extraction context. It goes beyond a strategy for syntax and depends on a semantic strategy. Industry 5.0 democratizes the co-production of information from Big Data, building on the current symmetrical innovation concept. The Industrial Revolution 5.0 is transforming companies into working through human and computer cooperation with the massive amount of data. Industry 5.0 develops human expertise and accuracy of the computer and will be creative and satisfy consumer needs with the final product. Big data produces usable data and analyzes the best data suited for the good of the industry. The industry has no benefit from NLP converted knowledge without Big Data. Currently, users can get information from several Web sites and do not have enough time to scan all Web sites. Data is distributed with various forms of data such as education, cinema, and politics. Numerous Web sites and social media (WhatsApp, Twitter, etc.) data are distributed in the world. These different data are collected via social media and stored in one SoloDB database that allows the user to access it quickly and easily. With the authorization of the administrative process, the database information can be accessed. Deep natural language, Big Data, and artificial intelligence will be discussed, and the results will be evaluated using the Industry 5.0 private database. The combination of computer and person would make it easy to access information from the database in a customized manner.

show abstract

Data Cleaning-A Thorough Analysis and Survey on Unstructured Data

Cited by 30 publications

References 2 publications

A Model for Enhancing Unstructured Big Data Warehouse Execution Time

A Model for Enhancing Unstructured Big Data Warehouse Execution Time

Voltaiq data format—A standard data format for collection of battery data to enable big data comparisons and analyses across the battery lifecycle

SoloDB for Social Media’s Big Data Using Deep Natural Language with AI Applications and Industry 5.0

Contact Info

Product

Resources

About