Representation biases in sentence transformers

Nikolaev, Dmitry; Padó, Sebastian

doi:10.18653/v1/2023.eacl-main.268

Cited by 3 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…La obra de Paul Chilton en "Analysing Political Discourse" (2004) ya advertía sobre las limitaciones inherentes a estos tipos de discursos (Chilton, 2004). En el mismo tenor, el estudio de Nikolaev et al (2023) sobre la estimación multilingüe del posicionamiento de partidos políticos ha corroborado estas deficiencias metodológicas (Nikolaev et al, 2023).…”

Section: El Análisis De Discursounclassified

See 1 more Smart Citation

Evolución del Discurso Presidencial sobre la democracia en Paraguay: un análisis comparativo

Benítez Cardozo,

Gómez Garozzo

2023

Rev. Juríd. Univ. Am.

View full text Add to dashboard Cite

Se analiza la evolución del concepto de democracia en los discursos de los presidentes de Paraguay desde 1989 hasta 2018. Se realizó un estudio documental utilizando 14 códigos agrupados en tres categorías, que permitió identificar patrones temáticos recurrentes en los discursos presidenciales y compararlos entre diferentes mandatos. Los discursos presidenciales son un aspecto fundamental para comprender la dinámica política y social del país. Se aborda la manera en que el concepto de democracia es utilizado por los presidentes en sus discursos. Esto es relevante para comprender cómo se define y promueve la democracia en el discurso político y su impacto en la sociedad. Los resultados demuestran que los presidentes paraguayos en sus discursos conceptúan una democracia pluralista, transparente, justa y participativa con instituciones fuertes, respetadas y con acceso a la educación,la justicia, la seguridad y el desarrollo económico. También enfatizan la lucha contra la corrupción, la desigualdad y la promoción de la integración regional. Este enfoque está inscripto en la línea de análisis de discursos, no en análisis de acciones o desempeño, pero es sabido que los discursos, y especialmente los presidenciales pueden tener implicaciones en la toma de decisiones gubernamentales, aunque no es el objetivo directo de este análisis.

show abstract

Section: El Análisis De Discursounclassified

“…Mario Abdo Benítez (2018-2023 Miembro del Partido Colorado y asumió la presidencia a través de elecciones democráticas. El padre de Abdo, Mario Abdo Benítez Sr., fue secretario privado de Stroessner y se le conocía como un cercano aliado del régimen.…”

unclassified

Evolución del Discurso Presidencial sobre la democracia en Paraguay: un análisis comparativo

Benítez Cardozo,

Gómez Garozzo

2023

Rev. Juríd. Univ. Am.

View full text Add to dashboard Cite

show abstract

“…Sentence transformers (Reimers and Gurevych, 2019) implement architecture and training regime changes on BERT to optimize sentence embeddings for downstream tasks. Nikolaev and Padó (2023) analyze the relation between specific sentence properties (e.g. the contribution of different POS) and the geometry of the embedding space of sentence transformers.…”

Section: Related Workmentioning

confidence: 99%

“…Specific grammatical phenomena are often studied on specifically designed or selected datasets (e.g. (Nikolaev and Padó, 2023;Linzen et al, 2016)). We use BLM-AgrF .…”

Section: Datamentioning

confidence: 99%

Grammatical information in BERT sentence embeddings as two-dimensional arrays

Năstase¹,

Merlo²

2023

Proceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023)

View full text Add to dashboard Cite

Sentence embeddings induced with various transformer architectures encode much semantic and syntactic information in a distributed manner in a one-dimensional array. We investigate whether specific grammatical information can be accessed in these distributed representations. Using data from a task developed to test rule-like generalizations, our experiments on detecting subject-verb agreement yield several promising results. First, we show that while the usual sentence representations encoded as onedimensional arrays do not easily support extraction of rule-like regularities, a two-dimensional reshaping of these vectors allows various learning architectures to access such information. Next, we show that various architectures can detect patterns in these two-dimensional reshaped sentence embeddings and successfully learn a model based on smaller amounts of simpler training data, which performs well on more complex test data. This indicates that current sentence embeddings contain information that is regularly distributed, and which can be captured when the embeddings are reshaped into higher dimensional arrays. Our results cast light on representations produced by language models and help move towards developing fewshot learning approaches.

show abstract

TransVAE-PAM: A Combined Transformer and DAG-based Approach for Enhanced Fake News Detection in Indian Context

Tufchi,

Ahmed,

Yadav

et al. 2024

ACM Trans. Asian Low-Resour. Lang. Inf. Process.

View full text Add to dashboard Cite

In this study, we introduce a novel method, “TransVAE-PAM”, for the classification of fake news articles, tailored specifically for the Indian context. The approach capitalizes on state-of-the-art contextual and sentence transformer-based embedding models to generate article embeddings. Furthermore, we also try to address the issue of compact model size. In this respect, we employ a Variational Autoencoder (VAE) and β -VAE to reduce the dimensions of the embeddings, thereby yielding compact latent representations. To capture the thematic essence or important topics in the news articles, we use the Pachinko Allocation Model (PAM) model, a Directed Acyclic Graph (DAG) based approach, to generate meaningful topics. These two facets of representation - the reduced-dimension embeddings from the VAE and the extracted topics from the PAM model - are fused together to create a feature set. This representation is subsequently channeled into five different methods for fake news classification. Furthermore, we use eight distinct transformer-based architectures to test the embedding generation. To validate the feasibility of the proposed approach, we have conducted extensive experimentation on a proprietary dataset. The dataset is sourced from “Times of India” and other online media. Considering the size of the dataset, large-scale experiments are conducted on an NVIDIA supercomputer. Through this comprehensive numerical investigation, we have achieved an accuracy of 96.2% and an F1 score of 96% using the DistilBERT transformer architecture. By complementing the method via topic modeling, we record a performance improvement with the accuracy and F1 score both at 97%. These results indicate a promising direction toward leveraging the combination of advanced topic models into existing classification schemes to enhance research on fake news detection.

show abstract

Representation biases in sentence transformers

Cited by 3 publications

References 14 publications

Evolución del Discurso Presidencial sobre la democracia en Paraguay: un análisis comparativo

Evolución del Discurso Presidencial sobre la democracia en Paraguay: un análisis comparativo

Grammatical information in BERT sentence embeddings as two-dimensional arrays

TransVAE-PAM: A Combined Transformer and DAG-based Approach for Enhanced Fake News Detection in Indian Context

Contact Info

Product

Resources

About