Skluma: An Extensible Metadata Extraction Pipeline for Disorganized Data

Skluzacek, Tyler J.; Kumar, Rohan; Chard, Ryan; Harrison, Galen; Beckman, Paul G.; Chard, Kyle; Foster, Ian

doi:10.1109/escience.2018.00040

Cited by 10 publications

(6 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These are flattened to Neo4j graph structures with extensible metadata management in the data lake, categorizing for kinds of data: raw data, metadata, additional semantics, and the data fragment identifiers. Skluma [136] GEMMS [64], [117] HANDLE [43] Data vault [57], [107] Diamantiniet al [34], [35], [36] Aurum [48] Ingestion Metadata modeling Sawadogoet et al [127] GOODS [67], [68] DS-Prox [3], [4], [5] KAYAK [90], [91] Nargesian et al [104] Ronin [110] Dataset organization…”

Section: Single Data Storementioning

confidence: 99%

See 1 more Smart Citation

Data Lakes: A Survey of Functions and Systems

Hai

Koutras

Quix

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

Data lakes are becoming increasingly prevalent for big data management and data analytics. In contrast to traditional 'schema-on-write' approaches such as data warehouses, data lakes are repositories storing raw data in its original formats and providing a common access interface. Despite the strong interest raised from both academia and industry, there is a large body of ambiguity regarding the definition, functions and available technologies for data lakes. A complete, coherent picture of data lake challenges and solutions is still missing. This survey reviews the development, architectures, and systems of data lakes. We provide a comprehensive overview of research questions for designing and building data lakes. We classify the existing approaches and systems based on their provided functions for data lakes, which makes this survey a useful technical reference for designing, implementing and deploying data lakes. We hope that the thorough comparison of existing solutions and the discussion of open research challenges in this survey will motivate the future development of data lake research and practice.

show abstract

Section: Single Data Storementioning

confidence: 99%

“…6.4, we further address extracting hidden metadata such as functional dependencies. Given semi-structured or unstructured data, existing approaches [53], [61], [117] extract primarily structural metadata, while the one in [136] extracts metadata related to content and context.…”

Section: Metadata Extractionmentioning

confidence: 99%

Data Lakes: A Survey of Functions and Systems

Hai

Koutras

Quix

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

show abstract

“…The Hadoop Distributed File System (HDFS) is one of the most frequently mentioned data storage systems for data lakes [21,119,13]. HDFS supports a wide range of DATAMARAN [48] Skluma [118] Metadata modeling Generic metadata model (GEMMS [108], Constance [59]) Data vault [98,52] Graph-based metadata model (Diamantiniet al [32,33], Aurum [43], Sawadogoet et al [113])…”

Section: File-based Storage Systemsmentioning

confidence: 99%

“…For scientific data files, Skluma [118] extracts JSON metadata for content and context of scientific data files. It first finds the name, path, size, and extension of the files; then it infers file types and adds specific extractors accordingly to process tabular data, free texts or null values, etc.…”

Section: Metadata Extractionmentioning

confidence: 99%

Data lake concept and systems: a survey

Hai¹,

Koutras²,

Quix³

et al. 2021

Preprint

View full text Add to dashboard Cite

Although big data has been discussed for some years, it still has many research challenges, especially the variety of data. It poses a huge difficulty to efficiently integrate, access, and query the large volume of diverse data in information silos with the traditional 'schema-on-write' approaches such as data warehouses. Data lakes have been proposed as a solution to this problem. They are repositories storing raw data in its original formats and providing a common access interface. This survey reviews the development, definition, and architectures of data lakes. We provide a comprehensive overview of research questions for designing and building data lakes. We classify the existing data lake systems based on their provided functions, which makes this survey a useful technical reference for designing, implementing and applying data lakes. We hope that the thorough comparison of existing solutions and the discussion of open research challenges in this survey would motivate the future development of data lake research and practice.

show abstract

“…However, despite the simpler API for metadata storage, an inherited limitation of key-value store is; lack of multiattribute or multi-dimensional search queries, which is a common trend in scientific applications and communities [2], [27], [30], [46]. Scientific data is largely unstructured and contains a lot of descriptive metadata in the form of keyvalue pair attributes [26], and retrieving the desired dataset usually depends on such multiple metadata attributes [30], [47]. Besides, such retrieval often includes additional tags and annotations in search query provided by scientists and applications [2], [26].…”

Section: ) Object Sharing Controlsmentioning

confidence: 99%

MOSIQS: Persistent Memory Object Storage With Metadata Indexing and Querying for Scientific Computing

et al. 2021

View full text Add to dashboard Cite

Scientific applications often require high-bandwidth shared storage to perform joint simulations and collaborative data analytics. Shared memory pools provide a chance to satisfy such needs. Recently, a high-speed network such as Gen-Z utilizing persistent memory (PM) offers an opportunity to create a shared memory pool connected to compute nodes. However, there are several challenges to use scientific applications on the shared memory pool directly such as scalability, failure-atomicity, and lack of scientific metadata-based search and query. In this paper, we propose MOSIQS, a persistent memory object storage framework with metadata indexing and querying for scientific computing. We design MOSIQS based on the key idea that memory objects on PM pool can live beyond the application lifetime and can become the sharing currency for applications and scientists. MOSIQS provides an aggregate memory pool atop an array of persistent memory devices to store and access memory objects to accelerate scientific computing. MOSIQS uses a lightweight persistent memory key-value store to manage the metadata of memory objects, which enables memory object sharing. To facilitate metadata search and query over millions of memory objects resident on memory pool, we introduce Group Split and Merge (GSM), a novel persistent index data structure designed primarily for scientific datasets. GSM splits and merges dynamically to minimize the query search space and maintains low query processing time while overcoming the index storage overhead. MOSIQS is implemented atop of PMDK. We evaluate the proposed approach on many-core server with an array of real PM devices. Experimental results show that MOSIQS gains a 100% write performance improvement and executes multi-attribute queries efficiently with 2.7× less index storage overhead offering significant potential to speed up scientific computing applications.

show abstract

Skluma: An Extensible Metadata Extraction Pipeline for Disorganized Data

Cited by 10 publications

References 22 publications

Data Lakes: A Survey of Functions and Systems

Data Lakes: A Survey of Functions and Systems

Data lake concept and systems: a survey

MOSIQS: Persistent Memory Object Storage With Metadata Indexing and Querying for Scientific Computing

Contact Info

Product

Resources

About