Tempura: Query Analysis with Structural Templates

Wu, Tongshuang; Wongsuphasawat, Kanit; Ren, Donghao; Patel, Kayur; DuBois, Chris

doi:10.1145/3313831.3376451

Cited by 9 publications

(3 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applying key phrases naively with an exact match would yield low coverage in the unlabeled data (especially for larger phrases). To get more coverage, at each iteration, we generalize key phrases extracted from labeled demonstrations into templates with combinations of tokens, lemmas, and part-of-speech tags [66,69], e.g.,"today" is expanded into today, NOUN, and DATE. We then select representative templates with a greedy weighted set coverage algorithm based on their specificity and the number of inputs they cover [59].…”

Section: Identifying Patterns With Key Phrase Clusteringmentioning

confidence: 99%

“…In a nutshell, ScatterShot helps users find informative input examples in the unlabeled data, annotate them efficiently with the help of the current version of the learned in-context function, and estimate the quality of said function. In each iteration, Scatter-Shot automatically slices the unlabeled data into clusters based on task-specific key phrases [66,69]. For example, given the existing examples in Figure 1A, it finds a cluster based on holiday key phrases ("Christmas", "Thanksgiving", etc.)…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ScatterShot: Interactive In-context Example Curation for Text Transformation

Weld

et al. 2023

Proceedings of the 28th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

The in-context learning capabilities of LLMs like GPT-3 allow annotators to customize an LLM to their specific tasks with a small number of examples. However, users tend to include only the most obvious patterns when crafting examples, resulting in underspecified in-context functions that fall short on unseen cases. Further, it is hard to know when "enough" examples have been included even for known patterns. In this work, we present ScatterShot, an interactive system for building high-quality demonstration sets for in-context learning. ScatterShot iteratively slices unlabeled data into task-specific patterns, samples informative inputs from underexplored or not-yet-saturated slices in an active learning manner, and helps users label more efficiently with the help of an LLM and the current example set. In simulation studies on two text perturbation scenarios, ScatterShot sampling improves the resulting few-shot functions by 4-5 percentage points over random sampling, with less variance as more examples are added. In a user study, ScatterShot greatly helps users in covering different patterns in the input space and labeling in-context examples more efficiently, resulting in better in-context learning and less user effort.

show abstract

Section: Identifying Patterns With Key Phrase Clusteringmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

ScatterShot: Interactive In-context Example Curation for Text Transformation

Weld

et al. 2023

Proceedings of the 28th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

show abstract

“…MSQ-NER: MS-MARCO Question NER To represent NER in the QA domain, we create a set of natural language questions, based on the MS-MARCO QnA corpus (V2.1)(Bajaj et al, 2016). LikeWu et al (2020), we templatize the questions by applying NER to extract item names, which are then mapped to our taxonomy. Entities are replaced with their types to create templates, e.g., "who…”

mentioning

confidence: 99%

GEMNET: Effective Gated Gazetteer Representations for Recognizing Complex Entities in Low-context Input

Meng¹,

Fang²,

Rokhlenko³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Named Entity Recognition (NER) remains difficult in real-world settings; current challenges include short texts (low context), emerging entities, and complex entities (e.g. movie names). Gazetteer features can help, but results have been mixed due to challenges with adding extra features, and a lack of realistic evaluation data. It has been shown that including gazetteer features can cause models to overuse or underuse them, leading to poor generalization. We propose GEMNET, a novel approach for gazetteer knowledge integration, including (1) a flexible Contextual Gazetteer Representation (CGR) encoder that can be fused with any word-level model; and (2) a Mixture-of-Experts gating network that overcomes the feature overuse issue by learning to conditionally combine the context and gazetteer features, instead of assigning them fixed weights. To comprehensively evaluate our approaches, we create 3 large NER datasets (24M tokens) reflecting current challenges. In an uncased setting, our methods show large gains (up to +49% F1) in recognizing difficult entities compared to existing baselines. On standard benchmarks, we achieve a new uncased SOTA on CoNLL03 and WNUT17.

show abstract