Understanding tables with intermediate pre-training

Eisenschlos, Julian Martin; Krichene, Syrine; Müller, Thomas

doi:10.18653/v1/2020.findings-emnlp.27

Cited by 46 publications

(16 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Depending on the focused downstream tasks, these models adopt different components, i.e., encoders or decoders, as summarized in Table 1. Most of them adopt the encoder part of transformers similar to BERT [30], including TURL [28], StruG [29], TAPAS [38,51], GraPPa [29],MATE [39], TUTA [95], ForTap [23], and TableFormer [6]. Typically, a single encoder is applied on the sequential inputs constructed from tables and associated texts, if any, to learn the contextual representations of the inputs.…”

Section: Encoder-decodermentioning

confidence: 99%

“…Unfortunately, large tables from webs, documents, and spreadsheets contain dozens of rows or columns, posing a significant challenge to the memory and computational efficiency [39]. A naive way [51,69] to deal with large tables is truncating the input tokens by a maximum sequence [38] ranks columns by Jaccard coefficient between the NL and each column tokens. The model is twice as fast to train as TaPas [51] while achieving similar performance.…”

Section: Model Efficiencymentioning

confidence: 99%

See 1 more Smart Citation

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

Dong¹,

Cheng²,

He³

et al. 2022

Preprint

View full text Add to dashboard Cite

Since a vast number of tables can be easily collected from web pages, spreadsheets, PDFs, and various other document types, a flurry of table pre-training frameworks have been proposed following the success of text and images, and they have achieved new state-of-thearts on various tasks such as table question answering, table type recognition, column relation classification, table search, formula prediction, etc. To fully use the supervision signals in unlabeled tables, a variety of pre-training objectives have been designed and evaluated, for example, denoising cell values, predicting numerical relationships, and implicitly executing SQLs. And to best leverage the characteristics of (semi-)structured tables, various tabular language models, particularly with specially-designed attention mechanisms, have been explored. Since tables usually appear and interact with free-form text, table pre-training usually takes the form of table-text joint pre-training, which attracts significant research interests from multiple domains. This survey aims to provide a comprehensive review of different model designs, pre-training objectives, and downstream tasks for table pre-training, and we further share our thoughts and vision on existing challenges and future opportunities.

show abstract

Section: Encoder-decodermentioning

confidence: 99%

Section: Model Efficiencymentioning

confidence: 99%

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

Dong¹,

Cheng²,

He³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Visualization Recommendation (VisRec) [189], [295] [33], [38], [43], [47], [53], [72], [86], [88], [107], [110], [115], [129], [137], [141], [148], [151], [153], [167], [169], [170], [187], [188], [205], [237], [250], [256], [264], [265], [266], [268], [283], [294] Natural Language Interface for DataBase (NLIDB) [6], [296] [13], [14], [15], [19], [22], [56], [74], [75], [76], [82], [89], [94], [96], [118], [134], [135],…”

Section: Topic Related To V-nli Survey Representative Papersmentioning

confidence: 99%

“…Furthermore, to capture the special format of SQL statement, SQLnet [273] adopts a universal sketch as template and predicts values for each slot, while [255] employs a two-stage pipeline to subsequently predict the semantic structure and generate SQL statement with structure-enhanced query text. Recently, TaPas [56], [82] extends BERT's architecture to encode tables as input and trains from weak supervision. During the development of the community, there also generated some benchmarks like WikiSQL [293] and Spider [281], which can be utilized for further V-NLI research.…”

Section: Nli For Databasementioning

confidence: 99%

Towards Natural Language Interfaces for Data Visualization: A Survey

Shen,

Luo

et al. 2021

Preprint

View full text Add to dashboard Cite

Utilizing Visualization-oriented Natural Language Interfaces (V-NLI) as a complementary input modality to direct manipulation for visual analytics can provide an engaging user experience. It enables users to focus on their tasks rather than worrying about operating the interface to visualization tools. In the past two decades, leveraging advanced natural language processing technologies, numerous V-NLI systems have been developed both within academic research and commercial software, especially in recent years. In this article, we conduct a comprehensive review of the existing V-NLIs. In order to classify each paper, we develop categorical dimensions based on a classic information visualization pipeline with the extension of a V-NLI layer. The following seven stages are used: query understanding, data transformation, visual mapping, view transformation, human interaction, context management, and presentation. Finally, we also shed light on several promising directions for future work in the community.

show abstract

“…For the second component of the task, we follow state-of-the-art entailment models (Zhou et al, 2019;Eisenschlos et al, 2020):…”

Section: Entailmentmentioning

confidence: 99%

Fool Me Twice: Entailment from Wikipedia Gamification

Eisenschlos¹,

Dhingra²,

Bulian³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

Self Cite

View full text Add to dashboard Cite

We release FOOLMETWICE (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, drastically lowering the number of examples that can be solved using "shortcuts" compared to other entailment datasets. Players are presented with two tasks. The first task asks the player to write a plausible claim based on the evidence from a Wikipedia page. The second one shows two plausible claims written by other players, one of which is false, and the goal is to identify it before the time runs out. Players "pay" to see clues retrieved from the evidence pool: the more evidence the player needs, the harder the claim. Game-play between motivated players leads to diverse strategies for crafting claims, such as temporal inference and diverting to unrelated evidence, and results in higher quality data for the entailment and evidence retrieval tasks. We open source the dataset and game code. 1 * Work completed while a Visiting Research Scientist at Google.

show abstract

Understanding tables with intermediate pre-training

Cited by 46 publications

References 37 publications

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

Towards Natural Language Interfaces for Data Visualization: A Survey

Fool Me Twice: Entailment from Wikipedia Gamification

Contact Info

Product

Resources

About