Distributing Text Mining tasks with
            <i>librAIry</i>

Badenes-Olmedo, Carlos; Redondo-García, José Luis; Corcho, Óscar

doi:10.1145/3103010.3121040

Cited by 8 publications

(4 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [14], an investigation of the influence of text length on IR tasks is shown using a Probabilistic Topic Model (PTM). The goal is to reduce the dimensions of the vectors in IR models to simplify the operations without affecting the system performance.…”

Section: Related Workmentioning

confidence: 99%

Recovering Scientific Literature on COVID-19 using level-based Text Analysis

Padilla-Cuevas

Cuevas-Rasgado

Reyes‐Ortiz

et al. 2023

Preprint

View full text Add to dashboard Cite

The pandemic caused by the SARS-CoV-2 virus has generated numerous scientific texts on the subject, including symptoms, drugs, treatments, and vaccines. These documents are written in natural language and lack a computer-processable structure, which causes a tedious and time-consuming manual analysis. Thus, Information Retrieval (IR) approaches are increasingly necessary to analyse Spanish scientific texts. Therefore, this paper presents approaches based on text analysis levels to retrieve relevant documents about COVID-19 from the Spanish literature. The approaches include preprocessing scientific texts, recovering relevant documents using text analysis levels (probabilistic, similarity, and semantic), and an evaluation process. The main aim of the approaches is to rank the scientific documents considering their relevance to the input question. The evaluation task is done with 100 COVID-19 questions and a dataset of 249,000 scientific texts in Spanish. The results show that the probabilistic approach achieved an F-measure of 85%, supported by document filtering using the Latent Dirichlet Allocation (LDA) algorithm.

show abstract

Section: Related Workmentioning

confidence: 99%

Recovering Scientific Literature on COVID-19 using level-based Text Analysis

Padilla-Cuevas

Cuevas-Rasgado

Reyes‐Ortiz

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Then, we set the number of topics 𝐾 = 500 (several configurations were evaluated, but this was the closest to the performance obtained with the supervised model based on categories). We run the Gibbs samplers for 1000 training iterations on LDA from the open-source librAIry [1] software. The Dirichlet priors 𝛼 = 0.1 and 𝛽 = 0.01 were set following [20].…”

Section: Cross-lingual Modelsmentioning

confidence: 99%

Scalable Cross-lingual Document Similarity through Language-specific Concept Hierarchies

Badenes-Olmedo,

García,

Corcho

2021

Preprint

Self Cite

View full text Add to dashboard Cite

With the ongoing growth in number of digital articles in a wider set of languages and the expanding use of different languages, we need annotation methods that enable browsing multi-lingual corpora. Multilingual probabilistic topic models have recently emerged as a group of semi-supervised machine learning models that can be used to perform thematic explorations on collections of texts in multiple languages. However, these approaches require theme-aligned training data to create a language-independent space. This constraint limits the amount of scenarios that this technique can offer solutions to train and makes it difficult to scale up to situations where a huge collection of multi-lingual documents are required during the training phase. This paper presents an unsupervised document similarity algorithm that does not require parallel or comparable corpora, or any other type of translation resource. The algorithm annotates topics automatically created from documents in a single language with cross-lingual labels and describes documents by hierarchies of multi-lingual concepts from independently-trained models. Experiments performed on the English, Spanish and French editions of JCR-Acquis corpora reveal promising results on classifying and sorting documents by similar content. CCS CONCEPTS• Information systems → Digital libraries and archives; Information retrieval.

show abstract

“…For each dataset, documents are mapped to two latent topic spaces with different dimensions using LDA. We perform parameter estimation using collapsed Gibbs sampling for LDA [16] from the open-source librAIry [4] software. It is a framework that combines natural language processing (NLP) techniques with machine learning algorithms on top of the Mallet toolkit [32], an open-source machine learning package.…”

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

Large-scale semantic exploration of scientific literature using topic-based hashing algorithms

Badenes-Olmedo

Redondo-García

Corcho

2020

Self Cite

View full text Add to dashboard Cite

Searching for similar documents and exploring major themes covered across groups of documents are common activities when browsing collections of scientific papers. This manual knowledge-intensive task can become less tedious and even lead to unexpected relevant findings if unsupervised algorithms are applied to help researchers. Most text mining algorithms represent documents in a common feature space that abstract them away from the specific sequence of words used in them. Probabilistic Topic Models reduce that feature space by annotating documents with thematic information. Over this low-dimensional latent space some locality-sensitive hashing algorithms have been proposed to perform document similarity search. However, thematic information gets hidden behind hash codes, preventing thematic exploration and limiting the explanatory capability of topics to justify content-based similarities. This paper presents a novel hashing algorithm based on approximate nearest-neighbor techniques that uses hierarchical sets of topics as hash codes. It not only performs efficient similarity searches, but also allows extending those queries with thematic restrictions explaining the similarity score from the most relevant topics. Extensive evaluations on both scientific and industrial text datasets validate the proposed algorithm in terms of accuracy and efficiency.

show abstract

Distributing Text Mining tasks with librAIry

Cited by 8 publications

References 6 publications

Recovering Scientific Literature on COVID-19 using level-based Text Analysis

Recovering Scientific Literature on COVID-19 using level-based Text Analysis

Scalable Cross-lingual Document Similarity through Language-specific Concept Hierarchies

Large-scale semantic exploration of scientific literature using topic-based hashing algorithms

Contact Info

Product

Resources

About