Do NLP Models Know Numbers? Probing Numeracy in Embeddings

Wallace, Eric; Wang, Yizhong; Li, Sujian; Singh, Sameer; Gardner, Matt

doi:10.18653/v1/d19-1534

Cited by 172 publications

(176 citation statements)

References 17 publications

Supporting

Mentioning

172

Contrasting

Order By: Relevance

“…First, that the OBESITY STATUS and OBESITY TYPES datasets contain much more numerical attributes (anthropometric information) than the SMOKING STATUS dataset. In this sense, it has been shown that BERT may not work properly representing numbers, while regular expressions allow representing complex sequential patterns, including numerical attributes [18], [22], [23]. Second, the SMOKING STATUS dataset presents temporal data and negations in the texts, and regular expressions need more examples to abstract the information.…”

Section: Discussionmentioning

confidence: 99%

Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions

2021

View full text Add to dashboard Cite

Biomedical text classification algorithms, which currently support clinical decision-making processes, call for expensive training texts due to the low availability of labeled corpus and the cost of manual annotation by specialized professionals. The active learning (AL) approach to classification heavily lessens such cost by reducing the number of labeled documents required to achieve specified performance. This article introduces a query strategy and a stopping criterion that transform CREGEX, a regular-expressions-based text classification algorithm, in an AL biomedical text classifier. The query strategy samples the training dataset, trading off the greedy learning achieved by the regular expressions classification precision and the conservative learning induced by text sequence alignment classification. The sustained reduction in the variance of the query strategy scores is used as a stopping criterion. The AL classifier was compared with Support Vector Machine (SVM), Naïve Bayes (NB), and a classifier based on Bidirectional Encoder Representations from Transformers (BERT), using three datasets with biomedical information in Spanish on smoking habits, obesity, and obesity types. The learning curve results indicate that AL in CREGEX allowed to efficiently reduce the number of training examples for equal performance than the rest of the classifiers, obtaining areas under the learning curve greater than 85% in all cases. The stopping criterion applied to the AL process allowed to use, on average, approximately 32% to 50% of the total training examples with differences in performance concerning the maximum value of the learning curve not exceeding 2%. This performance demonstrates the effectiveness of using AL in a biomedical text classifier based on regular expressions, which is attributable to such expressions' ability to represent intricate sequential patterns in training texts considered most informative.

show abstract

Section: Discussionmentioning

confidence: 99%

Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions

2021

View full text Add to dashboard Cite

show abstract

“…Within this context and drawing inspiration from [37], we propose a recursive approach that estimates the length of subsentence in an input sentence. We refer tp the length of subsentence as coverage, c, which is a scalar and computed as c = M N × κ, where M and N are the length of subsentence and input sentence, respectively.…”

Section: Coverage Model For Subsentence Extractionmentioning

confidence: 99%

Subsentence Extraction from Text Using Coverage-Based Deep Learning Language Models

Lim

Ahn

et al. 2021

Sensors

View full text Add to dashboard Cite

Sentiment prediction remains a challenging and unresolved task in various research fields, including psychology, neuroscience, and computer science. This stems from its high degree of subjectivity and limited input sources that can effectively capture the actual sentiment. This can be even more challenging with only text-based input. Meanwhile, the rise of deep learning and an unprecedented large volume of data have paved the way for artificial intelligence to perform impressively accurate predictions or even human-level reasoning. Drawing inspiration from this, we propose a coverage-based sentiment and subsentence extraction system that estimates a span of input text and recursively feeds this information back to the networks. The predicted subsentence consists of auxiliary information expressing a sentiment. This is an important building block for enabling vivid and epic sentiment delivery (within the scope of this paper) and for other natural language processing tasks such as text summarisation and Q&A. Our approach outperforms the state-of-the-art approaches by a large margin in subsentence prediction (i.e., Average Jaccard scores from 0.72 to 0.89). For the evaluation, we designed rigorous experiments consisting of 24 ablation studies. Finally, our learned lessons are returned to the community by sharing software packages and a public dataset that can reproduce the results presented in this paper.

show abstract

“…Recently, Kaushik et Numeric reasoning Numeric reasoning in Natural Language processing has been recognized as an important part in entailment models (Sammons et al, 2010) and reading comprehension (Ran et al, 2019). Wallace et al (2019) studied the capacity of different models on understanding numerical operations and show that BERT-based model still have headroom. This motivates the use of the synthetic generation approach to improve numerical reasoning in our model.…”

Section: Counterfactual Data Generationmentioning

confidence: 99%

Understanding tables with intermediate pre-training

Eisenschlos¹,

Krichene²,

Müller³

2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

Table entailment, the binary classification task of finding if a sentence is supported or refuted by the content of a table, requires parsing language and table structure as well as numerical and discrete reasoning. While there is extensive work on textual entailment, table entailment is less well studied. We adapt TAPAS (Herzig et al., 2020), a table-based BERT model, to recognize entailment. Motivated by the benefits of data augmentation, we create a balanced dataset of millions of automatically created training examples which are learned in an intermediate step prior to finetuning. This new data is not only useful for table entailment, but also for SQA (Iyyer et al., 2017), a sequential table QA task. To be able to use long examples as input of BERT models, we evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency at a moderate drop in accuracy. The different methods set the new state-of-the-art on the TAB-FACT (Chen et al., 2020) and SQA datasets.

show abstract

Do NLP Models Know Numbers? Probing Numeracy in Embeddings

Cited by 172 publications

References 17 publications

Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions

Active Learning for Biomedical Text Classification Based on Automatically Generated Regular Expressions

Subsentence Extraction from Text Using Coverage-Based Deep Learning Language Models

Understanding tables with intermediate pre-training

Contact Info

Product

Resources

About