Mining modern repositories with elasticsearch

Abstract

Organizations are generating, processing, and retaining data at a rate that often exceeds their ability to analyze it effectively; at the same time, the insights derived from these large data sets are often key to the success of the organizations, allowing them to better understand how to solve hard problems and thus gain competitive advantage. Because this data is so fast-moving and voluminous, it is increasingly impractical to analyze using traditional offline, read-only relational databases.Recently, new "big data" technologies and architectures, including Hadoop and NoSQL databases, have evolved to better support the needs of organizations analyzing such data. In particular, Elasticsearch -a distributed full-text search engine -explicitly addresses issues of scalability, big data search, and performance that relational databases were simply never designed to support. In this paper, we reflect upon our own experience with Elasticsearch and highlight its strengths and weaknesses for performing modern mining software repositories research.

A Survey Paper on Elastic Search Similarity Algorithm

“…To accomplish this necessity, elastic search spreads information to a few physical Lucene lists, these lists are called shards, and every one of the parts of the record is called sharding [1]. Elastic search can do this naturally, and every one of the parts of the record (shards) is unmistakable to the client as one major record.…”

“…and every one of the parts of the record is called sharding [1]. Elastic search can do this naturally, and every one of the parts of the record (shards) is unmistakable to the client as one major record.…”

“…The second sort of node is the primary node that functions as chief of the cluster controlling other nodes' work such as indexing. The third node sort is the tribe node, which is new and was presented in elastic search [1]. The tribe node can join numerous clusters and along these lines go about as an extension between them, permitting us to execute all elastic search functionalities on different groups.…”

A Distributed Framework for NLP-Based Keyword and Keyphrase Extraction From Web Pages and Documents

“…Therefore, applications and services must be able to scale up to items, domains and data subset of interest. Approaches based on parallel and distributed architectures are spreading also in search engines and indexing systems; for instance, the ElasticSearch 3 engine, which is an open source distributed search engine, designed to be scalable, near real-time capable and providing full-text search capabilities [5].…”

“…The present work describes a novel framework which allows the execution of general NLP tasks, through the use of the open source GATE 4 tool [11], on a multi-node cluster based on the open source Apache Hadoop 5 Distributed File system (HDFS). The paper is organized as follows: Section II illustrates related work, in terms of state of the art and open issues for both commercial and research literatures; in Section III, an architectural overview of the proposed system is presented; in Section IV, a validation of the system, performed against a real corpus of web resources, is reported; finally, Section V is left for conclusions and future perspectives.…”