cTBL: Augmenting Large Language Models for Conversational Tables

Abstract

An open challenge in multimodal conversational AI requires augmenting large language models with information from textual and nontextual sources for multi-turn dialogue. To address this problem, this paper introduces Conversational Tables (CTBLS), a three-step encoder-decoder architecture to retrieve tabular information and generate dialogue responses grounded on the retrieved information. CTBLS uses Transformer encoder embeddings for Dense Table Retrieval and obtains up to 5% relative improvement in Top-1 and Top-3 accuracy over sparse retrieval on the HYRBIDIALOGUE dataset. Additionally, CTBLS performs tabular knowledge retrieval using encoder and decoder models, resulting in up to 46% relative improvement in ROUGE scores and better human evaluation for response generation on HYRBIDIALOGUE.