This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of crosslingual transfer tasks. We train a Transformerbased masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing perlanguage performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code, data and models publicly available. 1

No abstract

No abstract

Over the past couple of years micro blogging platforms, such as Twitter, have become extremely popular for information generation and dissemination. Each day hundreds of millions of tweets are being published, containing fresh and trending information that is highly valuable for online users. However, discovering relevant information from such sources is becoming harder due to their rapid growth and the fact that social fragments are often short and noisy. Aggregation techniques such as clustering are often used for extracting this relevant information, since interesting signals begin to emerge when these fragments are grouped together. Clustering large amount of short tweets with limited features is however a challenging task in itself. In this paper, we propose to aggregate tweets by pivoting on entities and mapping them to topics that are already defined in websites such as Wikipedia and Freebase. This allows us to aggregate tweets in a more reliable and feasible way while providing interesting aggregated information about entities present in these fragments. Our analysis using large amounts of tweets shows that such an approach indeed works well. We present encouraging results and various interesting applications centered on entities.

Over the past couple of years, social networks such as Twitter and Facebook have become the primary source for consuming information on the Internet. One of the main differentiators of this content from traditional information sources available on the Web is the fact that these social networks surface individuals' perspectives. When social media users post and share updates with friends and followers, some of those short fragments of text contain a link and a personal comment about the web page, image or video. We are interested in mining the text around those links for a better understanding of what people are saying about the object they are referring to. Capturing the salient keywords from the crowd is rich metadata that we can use to augment a web page. This metadata can be used for many applications like ranking signals, query augmentation, indexing, and for organizing and categorizing content. In this paper, we present a technique called social signatures that given a link to a web page, pulls the most important keywords from the social chatter around it. That is, a high level representation of the web page from a social media perspective. Our findings indicate that the content of social signatures differs compared to those from a web page and therefore provides new insights. This difference is more prominent as the number of link shares increase. To showcase our work, we present the results of processing a dataset that contains around 1 Billion unique URLs shared in Twitter and Facebook over a two month period. We also provide data points that shed some light on the dynamics of content sharing in social media.