Probing Simile Knowledge from Pre-trained Language Models

Chen, Weijie; Chang, Yongzhu; Zhang, Rongsheng; Pu, Jiashu; Guandan, Chen,; Zhang, Le; Xi, Yadong; Chen, Yijiang; Su, Chang

doi:10.18653/v1/2022.acl-long.404

Cited by 16 publications

(31 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2) optimizing the step-1 model using the proximal policy optimization (PPO) (Schulman et al, 2017) method, the authors first built a comparison dataset by collecting responses from GPT-4, InstructGPT (Ouyang et al, 2022), and OPT-IML (Iyer et al, 2022) to a collection of instructions and then asked GPT-4 to rate each response from 1 to 10. Using the ratings, a reward model is trained based on OPT (Zhang et al, 2022a). The fine-tuned model from Step 1 is optimized by using the reward model to compute the policy gradient.…”

Section: Gpt-4-llmmentioning

confidence: 99%

“…OPT-IML (175B) (Iyer et al, 2022) is a large language model trained by fine-tuning the OPT (175B) (Zhang et al, 2022a) model on the constructed Instruction Meta-Learning (IML) dataset, which consists of over 1500 NLP tasks from 8 publicly available benchmarks such as PromptSource (Bach et al, 2022), FLAN (Longpre et al, 2023), and Super-NaturalInstructions (Wang et al, 2022d). After fine-tuning, OPT-IML outperforms OPT across all benchmarks.…”

Section: Othersmentioning

confidence: 99%

See 1 more Smart Citation

Applications of transformer-based language models in bioinformatics: a survey

Zhang

Fan

Liu

et al. 2023

Bioinformatics Advances

View full text Add to dashboard Cite

The Transformer-based language models, including vanilla Transformer, BERT, and GPT-3, have achieved revolutionary breakthroughs in the field of natural language processing. Since there are inherent similarities between various biological sequences and natural languages, the remarkable interpretability and adaptability of these models have prompted a new wave of their application in bioinformatics research. To provide a timely and comprehensive review, we introduce key developments of Transformer-based language models by describing the detailed structure of Transformers and summarize their contribution to a wide range of bioinformatics research from basic sequence analysis to drug discovery. While Transformer-based applications in bioinformatics are diverse and multifaceted, we identify and discuss the common challenges, including heterogeneity of training data, computational expense and model interpretability, and opportunities in the context of bioinformatics research. We hope that the broader community of natural language processing researchers, bioinformaticians, and biologists will be brought together to foster future research and development in Transformer-based language models, and inspire novel bioinformatics applications that are unattainable by traditional methods. Supplementary information Supplementary data are available at Bioinformatics Advances online.

show abstract

Section: Gpt-4-llmmentioning

confidence: 99%

Section: Othersmentioning

confidence: 99%

Applications of transformer-based language models in bioinformatics: a survey

Zhang

Fan

Liu

et al. 2023

Bioinformatics Advances

View full text Add to dashboard Cite

show abstract

“…It's worth noting that currently, open-source language models are larger and more diverse than vision models, making it easy to scale up our foundation models. For example, the largest OPT [98] has 175B parameters, while ViT-G [96] only has 1.8B.…”

Section: Progressive Pre-trainingmentioning

confidence: 99%

“…We use the UMT in both stages, significantly reducing the training sources and speeding up convergence. Thanks to readily-available image and language foundation models [62,59,53,98,16], our simple framework is easily scalable for video foundation models.…”

Section: Introductionmentioning

confidence: 99%

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

Li¹,

Wang²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scenerelated, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github. com/OpenGVLab/unmasked_teacher.

show abstract

“…Given the ML-ESG task's limited sample size (around 1200) and imbalanced label distribution, training on the available data alone is insufficient to fully train on 35 labels. To overcome these challenges, we employed three renowned open-source generative models: Pythia (Biderman et al, 2023), CerebrasGPT (Dey et al, 2023), and OPT (Zhang et al, 2022). Due to limitations in computational resources, we utilized a 12B model for Pythia, while CerebrasGPT and OPT utilized 13B models.…”

Section: Approachesmentioning

confidence: 99%