HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning

Abstract

Large crowdsourced datasets are widely used for training and evaluating neural models on natural language inference (NLI). Despite these efforts, neural models have a hard time capturing logical inferences, including those licensed by phrase replacements, socalled monotonicity reasoning. Since no large dataset has been developed for monotonicity reasoning, it is still unclear whether the main obstacle is the size of datasets or the model architectures themselves. To investigate this issue, we introduce a new dataset, called HELP, for handling entailments with lexical and logical phenomena. We add it to training data for the state-of-the-art neural models and evaluate them on test sets for monotonicity phenomena. The results showed that our data augmentation improved the overall accuracy. We also find that the improvement is better on monotonicity inferences with lexical replacements than on downward inferences with disjunction and modification. This suggests that some types of inferences can be improved by our data augmentation while others are immune to it.

Can Neural Networks Understand Monotonicity Reasoning?

“…These two figures show that, as the size of the upward training set increased, BERT performed better on upward inferences but worse on downward inferences, and vice versa. Previous work using HELP (Yanaka et al, 2019) reported that the BERT trained with MultiNLI and HELP containing both upward and downward inferences improved accuracy on both directions of monotonicity. MultiNLI rarely comes from downward inferences (see Section 4.3), and its size is large enough to be immune to the side-effects of downward inference examples in HELP.…”

“…To explore whether the performance of models on monotonicity reasoning depends on the training set or the model themselves, we conducted further analysis performed by data augmentation with the automatically generated monotonicity dataset HELP (Yanaka et al, 2019). HELP contains 36K monotonicity inference examples (7,784 upward examples, 21,192 downward examples, and 1,105 non-monotone examples).…”

“…In addition, we analyzed the performance of models trained with an automatically created monotonicity dataset, HELP (Yanaka et al, 2019). The analysis with monotonicity data augmentation shows that models tend to perform better in the same direction of monotonicity with the training set, while they perform worse in the opposite direction.…”

Exploring End-to-End Differentiable Natural Logic Modeling

“…We evaluate and analyze the proposed model on the monotonicity subset of Semantic Fragments (Richardson et al, 2020), HELP (Yanaka et al, 2019b) and MED (Yanaka et al, 2019a). We also extend MED to generate a dataset to help evaluate 2-hop inference.…”

“…Data: We use three datasets that are designed for studying monotonicity based reasoning, i.e., HELP (Yanaka et al, 2019b), MED (Yanaka et al, 2019a), and the monotonicity subset of Semantic Fragments (Richardson et al, 2020). The HELP dataset has 35,891 inference pairs, which are automatically generated by conducting lexical substitution or deletion on one sentence to obtain the other, given natural logic polarity information of each word token and syntactic structure of sentences.…”