Unstructured Data: an overview of the data of Big Data

Eberendu, Adanma Cecilia

doi:10.14445/22312803/ijctt-v38p109

Cited by 72 publications

(25 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…• Confidentiality [1]: Most of the custom datasets that the researchers have used, contain confidential data about people, administration, or business enterprises. For instance, invoices are private documents for any company or organization, containing information about the Buyer Name, GST Number, and few other fields.…”

Section: Privacy Issuesmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Layout Unstructured Invoice Documents Dataset: A Dataset for Template-Free Invoice Processing and Its Evaluation Using AI Approaches

2021

View full text Add to dashboard Cite

The daily transaction of an organization generates a vast amount of unstructured data such as invoices and purchase orders. Managing and analyzing unstructured data is a costly affair for the organization. Unstructured data has a wealth of hidden valuable information. Extracting such insights automatically from unstructured documents can significantly increase the productivity of an organization. Thus, there is a huge demand to develop a tool that can automate the extraction of key fields from unstructured documents. Researchers have used different approaches for extracting key fields, but the lack of annotated and highquality datasets is the biggest challenge. Existing work in this area has used standard and custom datasets for extracting key fields from unstructured documents. Still, the existing datasets face some serious challenges, such as poor-quality images, domain-related datasets, and a lack of data validation approaches to evaluate data quality. This work highlights the detailed process flow for endto-end key fields extraction from unstructured documents. This work presents a high-quality, multi-layout unstructured invoice documents dataset assessed with a statistical data validation technique. The proposed multi-layout unstructured invoice documents dataset is highly diverse in invoice layouts to generalize key field extraction tasks for unstructured documents. The proposed multilayout unstructured invoice documents dataset is evaluated with various feature extraction techniques such as Glove, Word2Vec, FastText, and AI approaches such as BiLSTM and BiLSTM-CRF. We also present the comparative analysis of feature extraction techniques and AI approaches on the proposed multi-layout unstructured invoice document dataset. We attained the best results with BiLSTM-CRF model. INDEX TERMS Artificial Intelligence (AI), information extraction, key field extraction, Named Entity Recognition (NER), template-free invoice processing, unstructured data.

show abstract

Section: Privacy Issuesmentioning

confidence: 99%

“…However, the efficient analysis of unstructured documents is challenging for almost 95% of business enterprises as unstructured data is not organized very well. It costs millions of dollars to analyze and integrate this data into the information management systems of the organization for a significant increase in productivity [1].…”

Section: Iintroductionmentioning

confidence: 99%

Multi-Layout Unstructured Invoice Documents Dataset: A Dataset for Template-Free Invoice Processing and Its Evaluation Using AI Approaches

2021

View full text Add to dashboard Cite

show abstract

“…Unstructured data is information that either does not have a predefined data model or is not organized in a pre-defined manner. It includes data types such as text, sound, image and video (Eberandu, 2016). It is the most difficult data to work with and needs to be transformed into structured form before it is processed.…”

Section: The Definitions Of Data Mining and Text Miningmentioning

confidence: 99%

Data Pre-processing in Text Mining

Aksoy¹,

Çelik²,

Gülseçen³

2020

Who Runs the World: Data

View full text Add to dashboard Cite

The fact that any kind of user has the ability to generate data with great ease at any time causes an increase in the importance of data mining. Considering the reality that the vast majority of the available data is composed of unstructured data and that the data in the text type is outnumbering, it proves the increasing interest in text mining and the abundance of studies in this field. However, in order to be able to examine an unstructured data type like text, which is quite different from machine language, it is necessary to make this data more structured and make the machine work. At this point, the data pre-processing step, which covers a large part of the entire text mining process, is of great importance. In this chapter, it is aimed to explain the text pre-processing phase on a basic level by supporting this using visuals. In doing so, it is primarily planned to focus on text mining and to explain in detail the characteristics of the data processed. In this context, it is aimed to explain the data pre-processing steps followed in order to overcome these difficulties by examining the difficulties created by the data in question. As a result, this chapter is a descriptive review of the data pre-processing phase in text mining, which covers some of the studies previously conducted on this subject.

show abstract

“…A data warehouse is a structured, nonvolatile historical storage system that makes data organized and oriented by subject for analytical purposes [1]. However, the data warehouse lacks the capacity to handle semi-structured and unstructured data, which is a typical feature of big data [2]. As a result, the concept of the data lake has been introduced into the business intelligence architecture as a big data storage solution, capable of serving as a heterogeneous data source for the data warehouse.…”

Section: Introductionmentioning

confidence: 99%

A generic metadata management model for heterogeneous sources in a data warehouse

et al. 2021

View full text Add to dashboard Cite

For more than 30 decades, data warehouses have been considered the only business intelligence storage system for enterprises. However, with the advent of big data, they have been modernized to support the variety and dynamics of data by adopting the data lake as a centralized data source for heterogeneous sources. Indeed, the data lake is characterized by its flexibility and performance when storing and analyzing data. However, the absence of schema on the data during ingestion increases the risk of the transformation of the data lake into a data swamp, so the use of metadata management is essential to exploit the data lake. In this paper, we will present a conceptual metadata management model for the data lake. Our solution will be based on a functional architecture of the data lake as well as on a set of features allowing the genericity of the metadata model. Furthermore, we will present a set of transformation rules, allowing us to translate our conceptual model into an owl ontology.

show abstract

Unstructured Data: an overview of the data of Big Data

Cited by 72 publications

References 0 publications

Multi-Layout Unstructured Invoice Documents Dataset: A Dataset for Template-Free Invoice Processing and Its Evaluation Using AI Approaches

Multi-Layout Unstructured Invoice Documents Dataset: A Dataset for Template-Free Invoice Processing and Its Evaluation Using AI Approaches

Data Pre-processing in Text Mining

A generic metadata management model for heterogeneous sources in a data warehouse

Contact Info

Product

Resources

About