Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Lyu, Xinxi; Min, Sewon; Beltagy, Iz; Zettlemoyer, Luke; Hajishirzi, Hannaneh

doi:10.18653/v1/2023.acl-long.129

Cited by 4 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to its simplicity yet effectiveness and versatileness across diverse tasks, several approaches have been introduced to improve the quality of the LLM context. To mention a few, Lyu et al (2023)…”

Section: Large Language Modelsmentioning

confidence: 99%

Measuring Phylogenetic Information of Incomplete Sequence Data

2021

View full text Add to dashboard Cite

Widely used approaches for extracting phylogenetic information from aligned sets of molecular sequences rely upon probabilistic models of nucleotide substitution or amino-acid replacement. The phylogenetic information that can be extracted depends on the number of columns in the sequence alignment and will be decreased when the alignment contains gaps due to insertion or deletion events. Motivated by the measurement of information loss, we suggest assessment of the Effective Sequence Length (ESL) of an aligned data set. The ESL can differ from the actual number of columns in a sequence alignment because of the presence of alignment gaps. Furthermore, the estimation of phylogenetic information is affected by model misspecification. Inevitably, the actual process of molecular evolution differs from the probabilistic models employed to describe this process. This disparity means the amount of phylogenetic information in an actual sequence alignment will differ from the amount in a simulated data set of equal size, which motivated us to develop a new test for model adequacy. Via theory and empirical data analysis, we show how to disentangle the effects of gaps and model misspecification. By comparing the Fisher information of actual and simulated sequences, we identify which alignment sites and tree branches are most affected by gaps and model misspecification.

show abstract

Section: Large Language Modelsmentioning

confidence: 99%

Measuring Phylogenetic Information of Incomplete Sequence Data

2021

View full text Add to dashboard Cite

show abstract

“…Previous work(Perez et al, 2021;Schick and Schütze, 2022;Bragg et al, 2021) have fully discussed the importance of true fewshot setting for ICL Min et al (2022a)Lu et al (2022);Lyu et al (2023) also try to improve ICL under this setting. Our approach for ICL examples reweighting is under true few-shot setting.7 ConclusionIn this paper, we find that treating demonstration examples equally can be a bottleneck for ICL, and assigning proper weights to examples can significantly improve performance.…”

mentioning

confidence: 99%

Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning

Yang,

Dai,

Wang

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Large Language Models (LLMs) have recently gained the In-Context Learning (ICL) ability with the models scaling up, allowing them to quickly adapt to downstream tasks with only a few demonstration examples prepended in the input sequence. Nonetheless, the current practice of ICL treats all demonstration examples equally, which still warrants improvement, as the quality of examples is usually uneven. In this paper, we investigate how to determine approximately optimal weights for demonstration examples and how to apply them during ICL. To assess the quality of weights in the absence of additional validation data, we design a masked self-prediction (MSP) score that exhibits a strong correlation with the final ICL performance. To expedite the weight-searching process, we discretize the continuous weight space and adopt beam search. With approximately optimal weights obtained, we further propose two strategies to apply them to demonstrations at different model positions. Experimental results on 8 text classification tasks show that our approach outperforms conventional ICL by a large margin. Our code are publicly available at https: github.com/Zhe-Young/WICL.

show abstract

Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis

Yang,

Kim,

Jang

et al. 2024

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Previous work in prompt engineering for large language models has introduced different gradient-free probability-based prompt selection methods that aim to choose the optimal prompt among the candidates for a given task but have failed to provide a comprehensive and fair comparison between each other. In this paper, we propose a unified framework to interpret and evaluate the existing probability-based prompt selection methods by performing extensive experiments on 13 common and diverse NLP tasks. We find that each of the existing methods can be interpreted as some variant of the method that maximizes mutual information between the input and the predicted output (MI). Utilizing this finding, we develop several other combinatorial variants of MI and increase the effectiveness of the oracle prompt selection method from 87.79% to 94.98%, measured as the ratio of the performance of the selected prompt to that of the optimal oracle prompt. Furthermore, considering that all the methods rely on the output probability distribution of the model that might be biased, we propose a novel calibration method called Calibration by Marginalization (CBM) that is orthogonal to the existing methods and helps increase the prompt selection effectiveness of the best method to 96.85%, achieving 99.44% of the oracle prompt F1 without calibration.1

show abstract

Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Cited by 4 publications

References 0 publications

Measuring Phylogenetic Information of Incomplete Sequence Data

Measuring Phylogenetic Information of Incomplete Sequence Data

Not All Demonstration Examples are Equally Beneficial: Reweighting Demonstration Examples for In-Context Learning

Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis

Contact Info

Product

Resources

About