Transformers and the Representation of Biomedical Background Knowledge

Wysocki, Oskar; Zhou, Zili; O’Regan, Paul; Ferreira, Deborah; Wysocka, Magdalena; Landers, Dónal; Freitas, André

doi:10.1162/coli_a_00462

Cited by 10 publications

(8 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cao et al [45] showed that predictions from MLM are highly prompt-biased and that, while external context apparently helps the models, its mainly via type guidance or answer leakage. In follow-up works, many domain-specific datasets and original prompting approaches were developed for factual knowledge extraction on MLM [46,47,48,49,50,51,17,52]. In the biomedical area, Sung et al [53] extended the work of Cao et al on a new dataset, bioLAMA, confirming their observations for biological knowledge.…”

Section: Related Workmentioning

confidence: 95%

See 1 more Smart Citation

Large Language Models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery

Wysocka

Wysocki

Delmas

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Background Inferring over and extracting information from Large Language Models (LLMs) trained on a large corpus of scientific literature can potentially drive a new era in biomedical research, reducing the barriers for accessing existing medical evidence. This work examines the potential of LLMs for dialoguing with biomedical background knowledge, using the context of antibiotic discovery as an exemplar motivational scenario. The context of biomedical discovery from natural products entails understanding the relational evidence between an organism (e.g. a Fungi such as Albifimbria verrucaria), an associated chemical (Verrucarin A) and its associated antibiotic properties (present antibiotic activity). Results This work provides a systematic assessment on the ability of LLMs to encode and express these relations, verifying for fluency, prompt-alignment, semantic coherence, factual knowledge and specificity of generated responses. The systematic analysis is applied to nine state-of-the-art models, from models specialised on biomedical scientific corpora to general models such as ChatGPT and GPT-4 in two prompting-based tasks: chemical compound definition generation and chemical compound-fungus relation determination. Results show that while recent models have improved in fluency, factual accuracy is still low and models are biased towards over-represented entities. The ability of LLMs to serve as biomedical knowledge bases is questioned, and the need for additional systematic evaluation frameworks is highlighted. The best performing GPT-4 produced a factual definition for 70% of chemical compounds and 43.6% factual relations to fungi, whereas the best open source model BioGPT-large 30% of the compounds and 30% of the relations for the best-performing prompt. Conclusions The results show that while LLMs are currently not fit for purpose to be used as biomedical factual knowledge bases, there is a promising emerging property in the direction of factuality as the models become domain specialised, scale-up in size and level of human feedback.

show abstract

Section: Related Workmentioning

confidence: 95%

“…Hallucinations tend to surface prominently in domain-specific discourse, including the sciences. This elicits the need to establish the ability of these models to encode and preserve factual knowledge as well as establishing methodologies for the critical evaluation of its representation properties [15,16,17].…”

Section: Introductionmentioning

confidence: 99%

Large Language Models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery

Wysocka

Wysocki

Delmas

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Using LLMs for cell biology: Beyond the large-scale models tailored for structured, non-linguistic data, recent initiatives have explored the direct manipulations of LLMs for biomedically-focused tasks. For example, Hou and Ji [18] employed ChatGPT for cell type annotation; Wysocki et al [19] probed biomedical information on BioBERT and BioMegatron embeddings; and Ye et al [20] utilized instruction fine-tuning to achieve competitive results on graph data task benchmarks with an LLM. While our paper is under preparation, Levine et al [21] has independently embarked on a conceptually related approach to ours, where each cell is transformed into a sequence of gene names, ranked by expression level and truncated at top 100 genes.…”

Section: Related Workmentioning

confidence: 99%

“…This approach allows for more nuanced and dynamic representations. For example, Hou and Ji [21] employed ChatGPT for cell type annotation; Wysocki et al [22] investigated biomedical meanings encoded by BioBERT and BioMegatron embeddings; and Ye et al [23] utilized instruction fine-tuning to achieve competitive results on graph data task benchmarks with an LLM. Compared to prior works that directly query LLMs for biological tasks, our method solely utilizes the input descriptions of each gene (which can be sourced from high-quality databases such as NCBI [24]) and the embedding model of LLMs, which suffers less from problems such as hallucination.…”

Section: Using Language Models For Cell Biologymentioning

confidence: 99%

GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT

Chen,

Zou

2023

Preprint

View full text Add to dashboard Cite

There has been significant recent progress in leveraging large-scale gene expression data to develop foundation models for single-cell transcriptomes such as Geneformer [1], scGPT [2], and scBERT [3]. These models infer gene functions and interrelations from the gene expression profiles of millions of cells, which requires extensive data curation and resource-intensive training. Here, we explore a much simpler alternative by leveraging ChatGPT embeddings of genes based on literature. Our proposal, GenePT, uses NCBI text descriptions of individual genes with GPT-3.5 to generate gene embeddings. From there, GenePT generates single-cell embeddings in two ways: (i) by averaging the gene embeddings, weighted by each gene's expression level; or (ii) by creating a sentence embedding for each cell, using gene names ordered by the expression level. Without the need for dataset curation and additional pretraining, GenePT is efficient and easy to use. On many downstream tasks used to evaluate recent single-cell foundation models --- e.g., classifying gene properties and cell types --- GenePT achieves comparable, and often better, performance than Geneformer and other methods. GenePT demonstrates that large language model embedding of literature is a simple and effective path for biological foundation models.

show abstract

“…After pretrained models such as BERT [10], ALBERT [11], XLNet [12], and ELECTRA [13] are proposed, they have been widely employed in various NLP tasks [14][15][16][17]. Many researches employed NER and RE models based on pretrained models and achieved good results.…”

Section: Related Workmentioning

confidence: 99%

Multigranularity Pruning Model for Subject Recognition Task under Knowledge Base Question Answering When General Models Fail

Wang,

Xu,

Song

et al. 2023

International Journal of Intelligent Systems

View full text Add to dashboard Cite

In general knowledge base question answering (KBQA) models, subject recognition (SR) is usually a precondition of finding an answer, and it is a common way to employ a general named entity recognition (NER) model such as BERT-CRF to recognize the subject. However, in previous researches, the difference between a NER task and a SR task is usually ignored, and a wrong entity recognized by the NER model will certainly lead to a wrong answer in the KBQA task, which is one bottleneck for KBQA performance. In this paper, a multigranularity pruning model (MGPM) is proposed to answer a question when general models fail to recognize a subject. In MGPM, the set of all possible subjects in the Knowledge Base (KB) is pruned by 4 multigranularity pruning submodels successively based on the constraint of relation (domain and tuple), string similarity, and semantic similarity. Experimental results show that our model is compatible with various KBQA models for both single-relation and complex questions answering. The integrated MGPM model (with the BERT-CRF model) achieves a SR accuracy of 94.4% on the SimpleQuestions dataset, 68.6% on the WebQuestionsSP dataset, and 63.7% on the WebQuestions dataset, which outperforms the original model by a margin of 3.6%, 8.6%, and 5.3%, respectively.

show abstract

Transformers and the Representation of Biomedical Background Knowledge

Cited by 10 publications

References 46 publications

Large Language Models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery

Large Language Models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery

GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT

Multigranularity Pruning Model for Subject Recognition Task under Knowledge Base Question Answering When General Models Fail

Contact Info

Product

Resources

About