TaPas: Weakly Supervised Table Parsing via Pre-training

Abstract

Answering natural language questions over tables is usually seen as a semantic parsing task. To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms. However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation. In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms. TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection. TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end. We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture. We additionally find that transfer learning, which is trivial in our setting, from WIK-ISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.

Understanding tables with intermediate pre-training

“…On SQA (Iyyer et al, 2017) a model pre-trained on the synthetic entailment data outperforms one pre-trained on the MASK-LM task alone (Table 3). Our best BERT Base model outpeforms the BERT-Large model of Herzig et al (2020) and a BERT-Large model trained on our data improves the previous state-of-the-art by 4 points on average question and sequence accuracy. See dev results and error bars in Appendix E. Efficiency As discussed in Section 3.3 and Appendix A.4, we can increase the model efficiency by reducing the input length.…”

“…We use a model architecture derived from BERT and add additional embeddings to encode the table structure, following the approach of Herzig et al (2020) to encode the input.…”

“…Token embeddings, position embeddings and segment embeddings are analogous to the ones used in standard BERT. Additionally we follow Herzig et al (2020) and use column and row embeddings which encode the two dimensional position of the cell that the token corresponds to and rank embeddings for numeric columns that encode the numeric rank of the cell with respect to the column, and provide a simple way for the model to know how a row is ranked according to a specific column.…”

“…We also use the SQA (Iyyer et al, 2017) dataset for pre-training (following Herzig et al (2020)) and for testing if our pre-training is useful for related tasks. SQA is a question answering dataset that was created by asking crowd workers to split a compositional subset of WikiTableQuestions (Pasupat and Liang, 2015) into multiple referential questions.…”

“…The current models are dominated by semantic parsing approaches that attempt to create logical forms from weak supervision. We, on the other hand, follow Herzig et al (2020) and and encode the tables with BERT-based models to directly predict the entailment decision. But while BERT models for text have been scrutinized and optimized for how to best pre-train and represent textual data, the same attention has not been applied to tabular data, limiting the effectiveness in this setting.…”