Sequence-to-sequence (seq2seq) models have achieved great success in semantic parsing tasks, but they tend to struggle on out-of-distribution (OOD) data. Despite recent progress, robust semantic parsing on large-scale tasks that combine challenges from both compositional generalization and natural language variations remains an unsolved issue. To encourage research in this area, this work introduces CUDON, a large-scale dialogue dataset in the Chinese language, specifically created to evaluate the compositional generalization of semantic parsing. The dataset contains about ten thousand multi-turn complex queries, and provides multiple splits with different degrees of train-test distribution divergence. We have investigated improving compositional generalization through grammar-based decoding on this dataset. With specially designed grammars that leverage program schema, we are able to significantly improve the accuracy of seq2seq semantic parsers on OOD splits: a LSTM-based parser using a Context-free Grammar (CFG) achieves over 25% higher accuracy than a standard seq2seq baseline; a parser using Tree-Substitution Grammar (TSG) improves parsing speed by five to seven times over the CFG parser with only a small accuracy loss. The grammar-based LSTM parsers also outperforms BART-and T5-based seq2seq parsers on the OOD splits, despite having less than one tenth of the parameters and no pretraining. We also validated our approach on the SMCalflow-CS dataset, specifically on the zero-shot learning task.

No abstract