UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models

Xie, Tianbao; Wu, Chen; Shi, Peng; Zhong, Ruiqi; Scholak, Torsten; Yasunaga, Michihiro; Wu, Chien-Sheng; Zhong, Ming; Yin, Pengcheng; Wang, Sida I.; Zhong, Victor; Wang, Bailin; Li, Chengzu; Boyle, Connor; Ni, Ansong; Yao, Zhiliang; Radev, Dragomir; Xiong, Caiming; Kong, Lingpeng; Zhang, Rui; Smith, Noah A.; Zettlemoyer, Luke; Chen, Yu

doi:10.48550/arxiv.2201.05966

Cited by 33 publications

(49 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Retrieval-augmented LMs. Several works (Lewis et al, 2020b;Xie et al, 2022) introduce a retrieval module for LMs, where given an anchor text (e.g. question), retrieved text is added to the same LM context to improve model inference (e.g.…”

Section: Related Workmentioning

confidence: 99%

LinkBERT: Pretraining Language Models with Document Links

Yasunaga¹,

Leskovec²,

Liang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Language model (LM) pretraining can learn various knowledge from text corpora, helping downstream tasks. However, existing methods such as BERT model a single document, and do not capture dependencies or knowledge that span across documents. In this work, we propose LinkBERT, an LM pretraining method that leverages links between documents, e.g., hyperlinks. Given a text corpus, we view it as a graph of documents and create LM inputs by placing linked documents in the same context. We then pretrain the LM with two joint self-supervised objectives: masked language modeling and our new proposal, document relation prediction. We show that LinkBERT outperforms BERT on various downstream tasks across two domains: the general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with citation links). LinkBERT is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our biomedical LinkBERT sets new states of the art on various BioNLP tasks (+7% on BioASQ and USMLE). We release our pretrained models, LinkBERT and BioLinkBERT, as well as code and data. 1 * Equal senior authorship. 1 Available at https://github.com/michiyasunaga/ LinkBERT.

show abstract

Section: Related Workmentioning

confidence: 99%

LinkBERT: Pretraining Language Models with Document Links

Yasunaga¹,

Leskovec²,

Liang³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…TableGPT [46] distinctly adapts a template-based table serialization way on relatively simple tables. Experiments conducted by UnifiedSKG [97] show that putting eternal text (like questions) ahead of tables helps T5 to generalise better on tabular tasks. [66] directly encodes markup languages like NL.…”

Section: Tabular Sequence Serializationmentioning

confidence: 99%

“…RPT [87] also adopts the encoder-decoder model, similar to a BERT [30] model combined with a GPT-3 [10] model. TaPEx [69] and Uni-fiedSKG [97] implements the encoder-decoder (text-to-text) model based on BART [64] and T5 [80], respectively, for downstream [46] directly fine-tunes pre-trained GPT-2 decoder to take advantage of its contextual knowledge learned from linguistic corpora.…”

Section: Encoder-decodermentioning

confidence: 99%

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

Dong¹,

Cheng²,

He³

et al. 2022

Preprint

View full text Add to dashboard Cite

Since a vast number of tables can be easily collected from web pages, spreadsheets, PDFs, and various other document types, a flurry of table pre-training frameworks have been proposed following the success of text and images, and they have achieved new state-of-thearts on various tasks such as table question answering, table type recognition, column relation classification, table search, formula prediction, etc. To fully use the supervision signals in unlabeled tables, a variety of pre-training objectives have been designed and evaluated, for example, denoising cell values, predicting numerical relationships, and implicitly executing SQLs. And to best leverage the characteristics of (semi-)structured tables, various tabular language models, particularly with specially-designed attention mechanisms, have been explored. Since tables usually appear and interact with free-form text, table pre-training usually takes the form of table-text joint pre-training, which attracts significant research interests from multiple domains. This survey aims to provide a comprehensive review of different model designs, pre-training objectives, and downstream tasks for table pre-training, and we further share our thoughts and vision on existing challenges and future opportunities.

show abstract

“…There are two series of related works: invasive methods and non-invasive methods. Invasive methods, which are built on a strong assumption that the inner structure (e.g., self-attention and feedforward layers) of the PLM can be modified, includes Prefix-Tuning (Li and Liang, 2021), Bitfit (Ben Zaken et al, 2021), Child-Tuning , P-Tuning v2 (Liu et al, 2021b), LoRA (Hu et al, 2021, UnifiedSKG (Xie et al, 2022) and Adapter-based models (Rebuffi et al, 2017;Houlsby et al, 2019;Lin et al, 2020;He et al, 2021;Pfeiffer et al, 2021). Non-invasive methods, which only modify input embeddings and regard the inner structure as a black box, mostly are prompting methods (including our Input-Tuning).…”

Section: Related Workmentioning

confidence: 99%

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

An¹,

Li²,

Lin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently the prompt-tuning paradigm has attracted significant attention. By only tuning continuous prompts with a frozen pretrained language model (PLM), prompt-tuning takes a step towards deploying a shared frozen PLM to serve numerous downstream tasks. Although prompt-tuning shows good performance on certain natural language understanding (NLU) tasks, its effectiveness on natural language generation (NLG) tasks is still underexplored. In this paper, we argue that one of the factors hindering the development of prompt-tuning on NLG tasks is the unfamiliar inputs (i.e., inputs are linguistically different from the pretraining corpus). For example, our preliminary exploration reveals a large performance gap between prompt-tuning and fine-tuning when unfamiliar inputs occur frequently in NLG tasks. This motivates us to propose input-tuning, which fine-tunes both the continuous prompts and the input representations, leading to a more effective way to adapt unfamiliar inputs to frozen PLMs. Our proposed input-tuning is conceptually simple and empirically powerful. Experimental results on seven NLG tasks demonstrate that input-tuning is significantly and consistently better than prompt-tuning. Furthermore, on three of these tasks, input-tuning can achieve a comparable or even better performance than fine-tuning.

show abstract

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models

Cited by 33 publications

References 24 publications

LinkBERT: Pretraining Language Models with Document Links

LinkBERT: Pretraining Language Models with Document Links

Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks

Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models

Contact Info

Product

Resources

About