Learned in Translation: Contextualized Word Vectors

McCann, Bryan; Bradbury, James T.; Xiong, Caiming; Socher, Richard

doi:10.48550/arxiv.1708.00107

Cited by 33 publications

(36 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results are consistent with the findings in the literature -45% " 55% for in the 5-category classification, and 85% " 92% in the 2-category classification(McCann et al, 2017;Socher et al, 2013).…”

supporting

confidence: 93%

Sentiment Analysis and Effect of COVID-19 Pandemic using College SubReddit Data

Yan¹,

Liu²

2021

Preprint

View full text Add to dashboard Cite

The COVID-19 pandemic has affected societies and human health and well-being in various ways. In this study, we collected Reddit data from 2019 (pre-pandemic) and 2020 (pandemic) from the subreddits communities associated with 8 universities, applied natural language processing (NLP) techniques, and trained graphical neural networks with social media data, to study how the pandemic has affected people's emotions and psychological states compared to the pre-pandemic era. Specifically, we first applied a pre-trained Robustly Optimized BERT pre-training approach (RoBERTa) to learn embedding from the semantic information of Reddit messages and trained a graph attention network (GAT) for sentiment classification. The usage of GAT allows us to leverage the relational information among the messages during training. We then applied subgroup-adaptive model stacking to combine the prediction probabilities from RoBERTa and GAT to yield the final classification on sentiment. With the manually labeled and model-predicted sentiment labels on the collected data, we applied a generalized linear mixed-effects model to estimate the effects of pandemic and online teaching on people's sentiment in a statistically significant manner. The results suggest the odds of negative sentiments in 2020 is 14.6% higher than the odds in 2019 (p-value ă 0.001), and the odds of negative sentiments are 41.6% higher with in-person teaching than with online teaching in 2020 (p-value " 0.037) in the studied population.

show abstract

supporting

confidence: 93%

Sentiment Analysis and Effect of COVID-19 Pandemic using College SubReddit Data

Yan¹,

Liu²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…For general vision, Ima-geNet [9] pre-training can greatly assist downstream tasks, such as object detection [1,18,46] and semantic segmentation [35]. Also in natural language processing, representations pre-trained on web-crawled corpus via Mask Language Model [10] achieves leading performance on machine translation [38] and natural language inference [8].…”

Section: Related Workmentioning

confidence: 99%

PointCLIP: Point Cloud Understanding by CLIP

Zhang¹,

Zhang²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, zero-shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP) have shown inspirational performance on 2D visual recognition, which learns to match images with their corresponding texts in open-vocabulary settings. However, it remains under explored that whether CLIP, pre-trained by large-scale imagetext pairs in 2D, can be generalized to 3D recognition. In this paper, we identify such a setting is feasible by proposing PointCLIP, which conducts alignment between CLIPencoded point cloud and 3D category texts. Specifically, we encode a point cloud by projecting it into multi-view depth maps without rendering, and aggregate the view-wise zeroshot prediction to achieve knowledge transfer from 2D to 3D. On top of that, we design an inter-view adapter to better extract the global feature and adaptively fuse the few-shot knowledge learned from 3D into CLIP pre-trained in 2D. By just fine-tuning the lightweight adapter in the few-shot settings, the performance of PointCLIP could be largely improved. In addition, we observe the complementary property between PointCLIP and classical 3D-supervised networks. By simple ensembling, PointCLIP boosts baseline's performance and even surpasses state-of-the-art models. Therefore, PointCLIP is a promising alternative for effective 3D point cloud understanding via CLIP under low resource cost and data regime. We conduct thorough experiments on widely-adopted ModelNet10, ModelNet40 and the challenging ScanObjectNN to demonstrate the effectiveness of PointCLIP. The code is released at https: //github.com/ZrrSkywalker/PointCLIP.

show abstract

“…Various early work focuses on pre-training word embeddings for downstream tasks, such as Word2Vec [24] and Glove [25]. To handle the polysemy problem of word embeddings, modern PLMs built on shallow neural networks are proposed, like CoVE [26] and ELMo [27], which can provide contextual word representations. With the introduction of Transformer [28] and the advance of distributed computing systems, PLMs built on deep neural networks have gradually appeared, such as GPT [1], BERT [3] and XLNet [2].…”

Section: Related Workmentioning

confidence: 99%

Knowledge Inheritance for Pre-trained Language Models

Qin¹,

Lin²,

Yi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent explorations of large-scale pre-trained language models (PLMs) such as GPT-3 have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, training a large-scale PLM requires tremendous amounts of computational resources, which is timeconsuming and expensive. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring the availability of many existing welltrained PLMs. To this end, we explore the question that how can previously trained PLMs benefit training larger PLMs in future. Specifically, we introduce a novel pre-training framework named "knowledge inheritance" (KI), which combines both self-learning and teacher-guided learning to efficiently train larger PLMs. Sufficient experimental results demonstrate the feasibility of our KI framework. We also conduct empirical analyses to explore the effects of teacher PLMs' pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI can well support lifelong learning and knowledge transfer 1 .

show abstract

Learned in Translation: Contextualized Word Vectors

Cited by 33 publications

References 0 publications

Sentiment Analysis and Effect of COVID-19 Pandemic using College SubReddit Data

Sentiment Analysis and Effect of COVID-19 Pandemic using College SubReddit Data

PointCLIP: Point Cloud Understanding by CLIP

Knowledge Inheritance for Pre-trained Language Models

Contact Info

Product

Resources

About