Development and implementation of accelerated methods of data access

Abstract

In this paper we address the problem of organizing a permission-conscious access to files and their metadata. A large amount of data is required to be stored and analyzed in medicine, bioinformatics, astronomy, and banking. For effective data storage, it is necessary to apply different technologies depending on the data. We have developed a system that consists of the following components: a relational DBMS to store user permissions and file metadata; an off-the-shelf distributed search engine to store activity logs; a distributed cache with permissions; a file system; a special application that interacts with all the respective components. We describe the overall architecture of our approach and discuss each component in detail. To offload database server and to speed up the permission checking we have devised a simple data caching scheme that is performed on an application server. The idea was to manually rewrite queries that are used by our application to check permissions. As a result, queries that use several tables request only non-cached tables from the database. We conclude our paper with the experimental evaluation of our system and the proposed caching approach.