A Scalable Document-Based Architecture for Text Analysis

Abstract

Abstract. Analyzing textual data is a very challenging task because of the huge volume of data generated daily. Fundamental issues in text analysis include the lack of structure in document datasets, the need for various preprocessing steps and performance and scaling issues. Existing text analysis architectures partly solve these issues, providing restrictive data schemas, addressing only one aspect of text preprocessing and focusing on one single task when dealing with performance optimization. Thus, we propose in this paper a new generic text analysis architecture, where document structure is flexible, many preprocessing techniques are integrated and textual datasets are indexed for efficient access. We implement our conceptual architecture using both a relational and a document-oriented database. Our experiments demonstrate the feasibility of our approach and the superiority of the document-oriented logical and physical implementation.

TLATR: Automatic Topic Labeling Using Automatic (Domain-Specific) Term Recognition

“…Previous works have proven that the combination between topic model algorithm and weighting schema is really dependent on the document dataset [52], [53]. The quality of topic modeling can be influenced by the length of documents in the corpus, the frequency of rare words, etc.…”

“…The textual datasets are processed and stored in a MongoDB database. The architecture used for processing and storing documents is presented in detail in [53].…”

Benchmarking Top-K Keyword and Top-K Document Processing with T${}^2$K${}^2$ and T${}^2$K${}^2$D${}^2$

“…We designed our benchmarks' queries by logging computational linguists working on a text analysis platform [12,13] on real-world data. After analyzing and clustering similar user queries, we ended up with 8 queries (4 for T 2 K 2 and 4 for T 2 K 2 D 2 ) that we consider generic enough to benchmark any similar system.…”

T $$^2$$ K $$^2$$ : The Twitter Top-K Keywords Benchmark

“…Hence, we propose arXiv:1709.04747v1 [cs.DB] 14 Sep 2017 in this paper the Twitter Top-K Keywords Benchmark (T 2 K 2 ), which features a real tweet dataset and queries with various complexities and selectivities. We designed T 2 K 2 to be somewhat generic, i.e., it can compare various weighting schemes, database logical and physical implementations and even text analytics platforms [18] in terms of computing efficiency. As a proof of concept of T 2 K 2 's relevance and genericity, we show how to implement the TF-IDF and Okapi BM25 weighting schemes, on one hand, and relational and document-oriented database instantiations, on the other hand.…”