TABBIE: Pretrained Representations of Tabular Data

Abstract

Existing work on tabular representationlearning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. While this joint pretraining improves tasks involving paired tables and text (e.g., answering questions about tables), we show that it underperforms on tasks that operate over tables without any associated text (e.g., populating missing cells). We devise a simple pretraining objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the-art on a suite of tablebased prediction tasks. Unlike competing approaches, our model (TABBIE) provides embeddings of all table substructures (cells, rows, and columns), and it also requires far less compute to train. A qualitative analysis of our model's learned cell, column, and row representations shows that it understands complex table semantics and numerical trends.

Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning

“…Such approaches can be applied on other datasets such as WikiTableQA (Pasupat and Liang, 2015), TabFact (Chen et al, 2019), Hy-bridQA (Chen et al, 2020b;Zayats et al, 2021;Oguz et al, 2020), OpenTableQA (Chen et al, 2021), ToTTo (Parikh et al, 2020, Turing Tables (Yoran et al, 2021) i.e. table to text generation tasks, LogicTable and (Chen et al, 2020a) and recently proposed tabular reasoning models proposed in TAPAS (Müller et al, 2021;Herzig et al, 2020), TaBERT (Yin et al, 2020), TABBIE (Iida et al, 2021), TabGCN (Pramanick and Bhattacharya, 2021) and RCI (Glass et al, 2021).…”

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

“…On the other hand, methods built on dynamic contextual embedding have also been explored. Many of them are tokenized using WrodPiece, encoded via the token vocabulary, and initialized by BERT [30], such as TaBERT [101], TaPas [51], MATE [39], StruG [29], TableFormer [6], TUTA [95], ForTaP [23], and TABBIE [55]. TURL [28] was initialized by TinyBERT [56] and additionally used an entity vocabulary.…”

“…Most works such as Tapas [51], MATE [39], TableFormer [6], TUTA [95] and TURL [28] performs in this way. TABBIE [55] linearizes tables by rows and columns separately. TableGPT [46] distinctly adapts a template-based table serialization way on relatively simple tables.…”

“…Tables are widely used to organize and present data in various document types and database systems such as webpages, spreadsheets, PDFs, presentation slides, and MySQL, and gain increasing attention from the research community. Following the success of large-scale pre-training in natural language (NL) and computer vision domains, a flurry of research works have been proposed to leverage unlabeled tables for self-supervised pre-training and achieve promising results in table type classification [92], cell type classification [45,92], table question answering (QA) [51,101], table search [94], entity linking [28], column type identification [15,47], table augmentation [28,55], formula prediction [23], etc. On the one hand, similar to NL that has already proved the success of large-scale pre-training, tables have dense semantics stored in textual headers, captions, and notes.…”

ReasonBERT: Pre-trained to Reason with Distant Supervision

“…Most prevalent approaches are variants of pre-trained language models such as BERT and RoBERTa (Liu et al, 2019b). More recently, self-supervised pre-training has also shown promising results on modalities other than plain text, such as tables (Herzig et al, 2020;Deng et al, 2020;Iida et al, 2021), knowledge bases (Zhang et al, 2019;Peters et al, 2019) and image-text (Su et al, 2020). Meanwhile, there has also been work that uses pre-training to accommodate specific needs of downstream NLP tasks, such as open-domain retrieval (Guu et al, 2020), representing and predicting spans of text (Joshi et al, 2020) and semantic parsing Deng et al, 2021).…”