Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Lyu, Xinxi; Min, Sewon; Beltagy, Iz; Zettlemoyer, Luke; Hajishirzi, Hannaneh

doi:10.48550/arxiv.2212.09865

Cited by 7 publications

(17 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the pseudo-inputs created by SELF-ICL is based on only one reference, i.e., the given test input, the generated pseudo-inputs are likely to be of great semantic similarity with that test input, and fail to capture the correct input space distribution. In such since it has been shown that models tend to copy the labels paired with inputs that are very similar to the test input, known as the copying effect (Lyu et al, 2022). With no guarantee for the correctness of SELF-ICL's pseudo-labels, the copying effect would potentially hurt the ICL performance.…”

Section: The Entanglement Of Input Space and Input-label Mappingmentioning

confidence: 99%

“…Recently, a series of studies has been proposed to shed lights on the inner working of ICL (Xie et al, 2021;Reynolds and McDonell, 2021;Min et al, 2022b). Their evidences suggest that instead of contributing explicit signals for learning new tasks, demonstrations mainly expose large LMs' intrinsic functionalities and guide models towards target domains (Razeghi et al, 2022;Lyu et al, 2022). Similar clues are also partly observed in chain-of-thought (CoT) (Madaan and Yazdanbakhsh, 2022) and instruction-augmented ICL (Webson and Pavlick, 2022).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A study of adults' orientations to learning at the national open university of Taiwan

Chen¹

View full text Add to dashboard Cite

Lifelogging has gained more attention due to its wide applications, such as personalized recommendations or memory assistance. The issues of collecting and extracting personal life events have emerged. People often share their life experiences with others through conversations. However, extracting life events from conversations is rarely explored. In this paper, we present Life Event Dialog, a dataset containing fine-grained life event annotations on conversational data. In addition, we initiate a novel conversational life event extraction task and differentiate the task from the public event extraction or the life event extraction from other sources like microblogs. We explore three information extraction (IE) frameworks to address the conversational life event extraction task: OpenIE, relation extraction, and event extraction. A comprehensive empirical analysis of the three baselines is established. The results suggest that the current event extraction model still struggles with extracting life events from human daily conversations. Our proposed life event dialog dataset and in-depth analysis of IE frameworks will facilitate future research on life event extraction from conversations.

show abstract

Section: The Entanglement Of Input Space and Input-label Mappingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A study of adults' orientations to learning at the national open university of Taiwan

Chen¹

View full text Add to dashboard Cite

show abstract

“…The performance of ICIL is comparable to adaptive in-context learning methods. We compare ICIL, which samples a fixed demonstration set for all evaluation tasks, with adaptive zero-shot in-context learning (Lyu et al, 2022), where the retrieved demonstrations vary based on the similarity of the target task or instance. Similar to ICIL, for adaptive zero-shot in-context learning, we retrieve the demonstrations which consist of instruction, input, and output instances from the training tasks of SUPERNI benchmark 8 .…”

Section: Additional Experimentsmentioning

confidence: 99%

“…However, Min et al (2022b) show that assigning random labels for demonstrations does not hurt the in-context learning performance much. Motivated by this finding, Lyu et al (2022) propose a zero-shot in-context learning method, retrieving relevant sentences from an external corpus and assigning random labels to construct demonstrations for classification target tasks. Different from Lyu et al (2022), ICIL utilizes instructions to facilitate task adaptation, uses a fixed set of demonstrations to evaluate all tasks, and is applicable to generation target tasks as well.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

In-Context Instruction Learning

Seonghyeon¹,

Hyeonbin²,

Yang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Instruction learning of Large Language Models (LLMs) has enabled zero-shot task generalization. However, instruction learning has been predominantly approached as a fine-tuning problem, including instruction tuning and reinforcement learning from human feedback, where LLMs are multi-task fine-tuned on various tasks with instructions. In this paper, we present a surprising finding that applying in-context learning to instruction learning, referred to as In-Context Instruction Learning (ICIL), significantly improves the zero-shot task generalization performance for both pretrained and instruction-fine-tuned models. One of the core advantages of ICIL is that it uses a single fixed prompt to evaluate all tasks, which is a concatenation of cross-task demonstrations. In particular, we demonstrate that the most powerful instruction-fine-tuned baseline (text-davinci-003) also benefits from ICIL by 9.3%, indicating that the effect of ICIL is complementary to instructionbased fine-tuning 2 . * Work done while interning at LG AI Research. 2 All experiments are reproducible from github.com/seonghyeonye/ICIL. Preprint. Under review.

show abstract

Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following

Ye,

Hwang,

Yang

et al. 2024

AAAI

View full text Add to dashboard Cite

In this paper, we present our finding that prepending a Task-Agnostic Prefix Prompt (TAPP) to the input improves the instruction-following ability of various Large Language Models (LLMs) during inference. TAPP is different from canonical prompts for LLMs in that it is a fixed prompt prepended to the beginning of every input regardless of the target task for zero-shot generalization. We observe that both base LLMs (i.e. not fine-tuned to follow instructions) and instruction-tuned models benefit from TAPP, resulting in 34.58% and 12.26% improvement on average, respectively. This implies that the instruction-following ability of LLMs can be improved during inference time with a fixed prompt constructed with simple heuristics. We hypothesize that TAPP assists language models to better estimate the output distribution by focusing more on the instruction of the target task during inference. In other words, such ability does not seem to be sufficiently activated in not only base LLMs but also many instruction-fine-tuned LLMs.

show abstract

Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Cited by 7 publications

References 0 publications

A study of adults' orientations to learning at the national open university of Taiwan

A study of adults' orientations to learning at the national open university of Taiwan

In-Context Instruction Learning

Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following

Contact Info

Product

Resources

About