Building a Query Engine for a Corpus of Open Data

Pelucchi, Mauro; Psaila, Giuseppe; Toccu, Maurizio

doi:10.5220/0006308801260136

Cited by 5 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, we briefly present Hammer, the framework for blind querying Open Data. This framework constitutes the background of the present work-it was introduced in Reference [2] and further improved in in Reference [3]; in this paper, we rely on the latter version [3].…”

Section: Blind Querying On Open Data: Definitions and Approachmentioning

confidence: 99%

“…In this paper, we adapt an approach for blind querying previously developed for querying Open Data Portals [2,3], which creates a catalog describing the meta-data of the data sets published by the open data portals. At query time, users express their queries in a blind way (i.e., irrespective of the actual and heterogeneous structures of data sets).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Blind Queries Applied to JSON Document Stores

Marrara

Pelucchi²,

Psaila

2019

Information

Self Cite

View full text Add to dashboard Cite

Social Media, Web Portals and, in general, information systems offer their own Application Programming Interfaces (APIs), used to provide large data sets concerning every aspect of day-by-day life. APIs usually provide data sets as collections of JSON documents. The heterogeneous structure of JSON documents returned by different APIs constitutes a barrier to effectively query and analyze these data sets. The adoption of NoSQL document stores, such as MongoDB, is useful for gathering these data sets, but does not solve the problem of querying the final heterogeneous repository. The aim of this paper is to provide analysts with a tool, named HammerJDB, that allows for blind querying collections of JSON documents within a NoSQL document database. The idea below is that users may know the application domain but it may be that they are not aware of the real structures of the documents stored in the database—the tool for blind querying tries to bridge the gap, by adopting a query rewriting mechanism. This paper is an evolution of a technique for blind querying Open Data portals and of its implementation within the Hammer framework, presented in some previous work. In this paper, we evolve that approach in order to query a NoSQL document database by evolving the Hammer framework into the HammerJDB framework, which is able to work on MongoDB databases. The effectiveness of the new approach is evaluated on a data set (derived from a real-life one), containing job-vacancy ads collected from European job portals.

show abstract

Section: Blind Querying On Open Data: Definitions and Approachmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Blind Queries Applied to JSON Document Stores

Marrara

Pelucchi²,

Psaila

2019

Information

Self Cite

View full text Add to dashboard Cite

show abstract

“…We started our research on blind querying of Open Data corpora with [1]; the technique has been subsequently enhanced in [2]; this latter paper is the basis for the implementation of the Hammer prototype discussed in this paper. We previously discussed how to use Map-Reduce in the various components of the Hammer prototype in [3]; however, a deep analysis of performance was not performed.…”

Section: Related Workmentioning

confidence: 99%

“…In [1], we started a research project whose aim is to develop a technique for blind querying corpora of Open Data. The idea of blind querying is motivated by the fact that a corpus of Open Data possibly contains thousands of data sets, each one with its own structure that is unknown to users wishing to look for interesting data sets.…”

Section: Introductionmentioning

confidence: 99%

“…The retrieval technique introduced in [1] has been further extended and improved in [2]; in both papers, we discussed the effectiveness of the technique in terms of recall and precision. In particular, we compared our technique with Apache Solr; it is based on Apache Lucene like ElasticSearch: both are feature-rich search engines and more or less give the same performance; however, Apache Solr is recommended for text-oriented search engines and Elasticsearch is better to handle analytical queries.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hadoop vs. Spark: Impact on Performance of the Hammer Query Engine for Open Data Corpora

Pelucchi¹,

Psaila

Toccu

2018

Algorithms

Self Cite

View full text Add to dashboard Cite

The Hammer prototype is a query engine for corpora of Open Data that provides users with the concept of blind querying. Since data sets published on Open Data portals are heterogeneous, users wishing to find out interesting data sets are blind: queries cannot be fully specified, as in the case of databases. Consequently, the query engine is responsible for rewriting and adapting the blind query to the actual data sets, by exploiting lexical and semantic similarity. The effectiveness of this approach was discussed in our previous works. In this paper, we report our experience in developing the query engine. In fact, in the very first version of the prototype, we realized that the implementation of the retrieval technique was too slow, even though corpora contained only a few thousands of data sets. We decided to adopt the Map-Reduce paradigm, in order to parallelize the query engine and improve performances. We passed through several versions of the query engine, either based on the Hadoop framework or on the Spark framework. Hadoop and Spark are two very popular frameworks for writing and executing parallel algorithms based on the Map-Reduce paradigm. In this paper, we present our study about the impact of adopting the Map-Reduce approach and its two most famous frameworks to parallelize the Hammer query engine; we discuss various implementations of the query engine, either obtained without significantly rewriting the algorithm or obtained by completely rewriting the algorithm by exploiting high level abstractions provided by Spark. The experimental campaign we performed shows the benefits provided by each studied solution, with the perspective of moving toward Big Data in the future. The lessons we learned are collected and synthesized into behavioral guidelines for developers approaching the problem of parallelizing algorithms by means of Map-Reduce frameworks.

show abstract