TextBenDS: a Generic Textual Data Benchmark for Distributed Systems

Abstract

Extracting top-k keywords and documents using weighting schemes are popular techniques employed in text mining and machine learning for different analysis and retrieval tasks. The weights are usually computed in the data preprocessing step, as they are costly to update and keep track of all the modifications performed on the dataset. Furthermore, computation errors are introduced when analyzing only subsets of the dataset. Therefore, in a Big Data context, it is crucial to lower the runtime of computing weighting schemes, without hindering the analysis process and the accuracy of the machine learning algorithms. To address this requirement for the task of top-k keywords and documents, it is customary to design benchmarks that compare weighting schemes within various configurations of distributed frameworks and database management systems. Thus, we propose a generic document-oriented benchmark for storing textual data and constructing weighting schemes (TextBenDS). Our benchmark offers a generic data model designed with a multidimensional approach for storing text documents. We also propose using aggregation queries with various complexities and selectivities for constructing term weighting schemes, that are utilized in extracting top-k keywords and documents. We evaluate the computing performance of the queries on several distributed environments set within the Apache Hadoop ecosystem. Our experimental results provide interesting insights. As an example, MongoDB proves to have the best overall performance, while Spark's execution time remains almost the same, regardless of the weighting schemes.

Benchmarking Data Lakes Featuring Structured and Unstructured Data with DLBench

“…Admittedly, BigDataBench's Grep and TextBenDS's Top-K documents operation are relevant for data search. Similarly, Top-K keywords and WordCount are relevant to assess documents aggregation [9,22]. However, other operations such as finding most similar documents or clustering documents should also be considered.…”

“…DLBench's data model also differs from most big data benchmarks as it provides raw tabular files, inducing an additional data integration challenge. Moreover, DLBench includes a set of long textual documents that induces a different challenge than short texts such as tweets [22] and Wikipedia articles [23]. Finally, DLBench is data-centric, unlike big data benchmarks that focus on a particular technology, e.g., TPCx-HS and HiBench.…”

“…One of the latest purely textual benchmarks is TextBenDS [22], which aims to evaluate performances of text analysis and processing systems. For this purpose, TextBenDS proposes a tweet-based data model and two types of workloads, namely Top-K keywords and Top-K documents operations.…”

Breakthroughs on Cross-Cutting Data Management, Data Analytics, and Applied Data Science

“…efficient data models, data processing pipelines and architectures to integrate standard and big data sources (Jovanovic et al 2020) as well as to improve resource utilization and aggregate performance in shared environments (Michiardi et al 2020); predictive analytics to forecast product demand in the fashion industry (Gardino et al 2020) and techniques to deal with the lack of annotated data for sensor-based human activity recognition (Prabono et al 2020); text data processing to assess the performance of text storage systems through a generic benchmark (Truicȃ et al 2020) and innovative solutions to deal with specific use cases such as the legal domain (Bordino et al 2020); novel approaches for mining social media to support intelligent transportation systems (Vallejos et al 2020) and digging deep the IoT scenario (Ustek-Spilda et al 2020); -solutions to deal with privacy issues in distance learning systems (Preuveneers et al 2020).…”

“…Paper (Truicȃ et al 2020) proposes a generic benchmark, called TextBenDS, for assessing the performance of text storage systems. At the conceptual level, the benchmark models text data (documents) as a cube.…”