Design of a vertical search engine for synchrotron data: a big data approach using Hadoop ecosystem

Abstract

A synchrotron as an experimental physics facility can provide the opportunity of a multi-disciplinary research and collaboration between scientists in various fields of study such as physics, chemistry, etc. During the construction and operation of such facility valuable data regarding the design of the facility, instruments and conducted experiments are published and stored. It takes researchers a long time going through different results from generalized search engines to find their needed scientific information so that the design of a domain specific search engine can help researchers to find their desired information with greater precision. It also provides the opportunity to use the crawled data to create a knowledgebase and also to generate different datasets required by the researchers. There have been several other vertical search engines that are designed for scientific data search such as medical information. In this paper we propose the design of such search engine on top of the Apache Hadoop framework. Usage of Hadoop ecosystem provides the necessary features such as scalability, fault tolerance and availability. It also abstracts the complexities of search engine design by using different open source tools as building blocks, among them Apache Nutch for the crawling block and Apache Solr for indexing and query processing. Our primary results obtained by implementing the proposed method in single node mode, the index of over a hundred thousand pages was created with the average fetch interval of 30 days having 28 segments and approximately 570 MB size. The performance factors such as the usage of available bandwidth and system load were logged using Linux's sysstat package.

MOOCs One-Stop Shop: A Realization of a Unified MOOCs Search Engine

“…Hence, the initial recommendations were generated by build-in clustering components instead of by applying a recommender system. These applied techniques could be vital in achieving later-on personalization of future work [52]. Moreover, users made positive observations about the system's trials.…”

“…Thus, search engines are continuing to retrieve some irrelevant results. This forces the user to traverse many pages to reach the desired results, and he/she might have difficulty in locating content relevant to the initial query [52]. In contrast, recommender systems suffer from the cold-start phenomenon, which occurs in the absence of user information in the case of new users or users with only a few preferences [53], [54].…”