ParaText

Abstract

Automated analysis of unstructured text documents (e.g., web pages, newswire articles, research publications, business reports) is a key capability for solving important problems in areas including decision making, risk assessment, social network analysis, intelligence analysis, scholarly research and others. However, as data sizes continue to grow in these areas, scalable processing, modeling, and semantic analysis of text collections becomes essential. In this paper, we present the ParaText text analysis engine, a distributed memory software framework for processing, modeling, and analyzing collections of unstructured text documents. Results on several document collections using hundreds of processors are presented to illustrate the flexibility, extensibility, and scalability of the the entire process of text modeling from raw data ingestion to application analysis.

Application of formal grammar in text mining and construction of an ontology

“…Many works apply the Term-Frequency and Inverse Document Frequency (TF-IDF) method [4,7,10,23,24]. It is based on the calculation of the similarity of two documents on the basis of frequency of matching terms occurring in the two documents.…”