scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis

Abstract

Different from a recent approach to building the Foundation Models (FMs) for analyzing single-cell data based on the pre-training and fine-tuning framework, in this manuscript, we extend the concept from GenePT [1] and propose a novel approach to leverage the advantages from Large Language Models (LLMs) to formalize a foundation model for single-cell data analysis, known as scELMo. We utilize LLMs like GPT 3.5 as a generator for both the description of metadata information and the embeddings for such description. Moreover, we also combine the embeddings from LLMs with the raw data under both zero-shot learning framework and fine-tuning framework to handle different tasks. We proved that scELMo is capable of cell clustering, batch effect correction, and cell-type annotation without training a new model. Moreover, the fine-tuning framework of scELMo can help with more challenging tasks including in-silico treatment analysis or modeling the perturbation information. scELMo has a lighter structure and lower requirement for resources but impressive performance, which poses a promising path for developing domain-specific FMs.