One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Hongjin, Su,; Shi, Weijia; Kasai, Jungo; Wang, Yizhong; Hu, Yushi; Ostendorf, Mari; Yih, Wen-tau; Smith, Noah A.; Zettlemoyer, Luke; Chen, Yu

doi:10.48550/arxiv.2212.09741

Cited by 14 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Contrastive learning has also been adopted as a pre-training objective for sentence representation learning. (Wang et al 2022b;Su et al 2022) Compared to contrastive learning, generative learning approaches are less investigated in the field of self-supervised sentence representation learning. Generative sentence representation learning attempts to generate original sentences from their corrupted or masked version (Yang et al 2020;Wang, Reimers, and Gurevych 2021).…”

Section: Self-supervised Learningmentioning

confidence: 99%

DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning

Wang,

He,

Wang

et al. 2024

AAAI

View full text Add to dashboard Cite

Contrastive-learning-based methods have dominated sentence representation learning. These methods regularize the representation space by pulling similar sentence representations closer and pushing away the dissimilar ones and have been proven effective in various NLP tasks, e.g., semantic textual similarity (STS) tasks. However, it is challenging for these methods to learn fine-grained semantics as they only learn from the inter-sentence perspective, i.e., their supervision signal comes from the relationship between data samples. In this work, we propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks, standing up well in comparison to contrastive-learning-based methods. Notably, the proposed intra-sentence denoising objective complements existing inter-sentence contrastive methodologies and can be integrated with them to further enhance performance. Our code is available at https://github.com/xinghaow99/DenoSent.

show abstract

Section: Self-supervised Learningmentioning

confidence: 99%

DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning

Wang,

He,

Wang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Our Mistral 7B and LLaMA 2 models use InstructorEmbedding [43] for embedding text. The instruction we give is "Represent the scientific text:" as we are working only with scientific texts or documents.…”

Section: Recommendation Modulementioning

confidence: 99%

Real-Time Ideation Analyzer and Information Recommender

Blazevic,

Sina,

Secco

et al. 2024

Electronics

View full text Add to dashboard Cite

The benefits of ideation for both industry and academia alike have been outlined by countless studies, leading to research into various approaches attempting to add new ideation methods or examine how the quality of the ideas and solutions created can be measured. Although AI-based approaches are being researched, there is no attempt to provide the ideation participants with information that inspire new ideas and solutions in real time. Our proposal presents a novel and intuitive approach that supports users in real time by providing them with relevant information as they conduct ideation. By analyzing their ideas within the respective ideation sessions, our approach recommends items of interest with high contextual similarity to the proposed ideas, allowing users to skim through, for example, publications and inspire new ideas quickly. The recommendations also evolve in real time. As more ideas are written during the ideation session, the recommendations become more precise. This real-time approach is instantiated with various ideation methods as a proof of concept, and various models are evaluated and compared to identify the best model for working with ideas.

show abstract

“…Using the transformer-based technique, specifically the IN-STRUCTOR (Su et al 2022) model, we generate embeddings 8 of 768 dimensions with a float 32-bit data type. This state-of-the-art model is optimized for diverse downstream tasks, for example, classification, clustering, and semantic similarity.…”

Section: Data Embeddingsmentioning

confidence: 99%

iDRAMA-Scored-2024: A Dataset of the Scored Social Media Platform from 2020 to 2023

Patel,

Paudel,

De Cristofaro

et al. 2024

ICWSM

View full text Add to dashboard Cite

Online web communities often face bans for violating platform policies, encouraging their migration to alternative platforms. This migration, however, can result in increased toxicity and unforeseen consequences on the new platform. In recent years, researchers have collected data from many alternative platforms, indicating coordinated efforts leading to offline events, conspiracy movements, hate speech propagation, and harassment. Thus, it becomes crucial to characterize and understand these alternative platforms. To advance research in this direction, we collect and release a large-scale dataset from Scored -- an alternative Reddit platform that sheltered banned fringe communities, for example, c/TheDonald (a prominent right-wing community) and c/GreatAwakening (a conspiratorial community). Over four years, we collected approximately 57M posts from Scored, with at least 58 communities identified as migrating from Reddit and over 950 communities created since the platform's inception. Furthermore, we provide sentence embeddings of all posts in our dataset, generated through a state-of-the-art model, to further advance the field in characterizing the discussions within these communities. We aim to provide these resources to facilitate their investigations without the need for extensive data collection and processing efforts.

show abstract

One Embedder, Any Task: Instruction-Finetuned Text Embeddings

Cited by 14 publications

References 0 publications

DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning

DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning

Real-Time Ideation Analyzer and Information Recommender

iDRAMA-Scored-2024: A Dataset of the Scored Social Media Platform from 2020 to 2023

Contact Info

Product

Resources

About