Parameter-Efficient Fine-Tuning Enhances Adaptation of Single Cell Large Language Model for Cell Type Identification

Abstract

Single-cell sequencing transformed biology and medicine, providing an unprecedented high-resolution view at the cellular level. However, the vast variability inherent in single-cell sequencing data impedes its utility for in-depth downstream analysis. Inspired by the foundation models in natural language processing, recent advancements have led to the development of single-cell Large Language Models (scLLMs). These models are designed to discern universal patterns across diverse single-cell datasets, thereby enhancing the signal-to-noise ratio. Despite their potential, multiple studies indicate existing scLLMs do not perform well in zero-short settings, highlighting a pressing need for more effective adaptation techniques. This research proposes several adaptation techniques for scLLMs by preserving the original model parameters while selectively updating newly introduced tensors. This approach aims to overcome the limitations associated with traditional fine-tuning practices, such as catastrophic forgetting and computational inefficiencies. We introduce two Parameter-Efficient Fine-Tuning (PEFT) strategies specifically tailored to refine scLLMs for cell type identification. Our investigations utilizing scGPT demonstrate that PEFT can enhance performance, with the added benefit of up to a 90% reduction in parameter training compared to conventional fine-tuning methodologies. This work paves the way for a new direction in leveraging single-cell models with greater efficiency and efficacy in single-cell biology.

Spatial Deconvolution of Cell Types and Cell States at Scale Utilizing TACIT

“…Unsupervised clustering methods like Louvain produce inconsistent results based on resolution, UMAP dimensionality, or programming language (R vs. Python) 38 . Finding optimal parameters is time-consuming and often suboptimal 24 . In contrast, TACIT automates cell type annotation, mimicking manual gating with enhanced scalability and precision.…”

“…Even with extensive parameter tuning combined with multi-step clustering to identify cell populations of interest, the desired results remain elusive 21,22 . Deep learning algorithms are increasingly utilized in spatial omics for cell type identification, but they require comprehensive and diverse training data to improve their accuracy and applicability to handle the complexities of spatial multiomics 23,24 .…”

Spatial Deconvolution of Cell Types and Cell States at Scale Utilizing TACIT

“…Even with extensive parameter tuning combined with multi-step clustering to identify cell populations of interest, the desired results remain elusive 21,22 . Deep learning algorithms are increasingly utilized in spatial ‘omics for cell type identification, but it requires comprehensive and diverse training data to improve the accuracy and applicability of deep learning models in handling the complexities of spatial multiomics 23,24 .…”

Nicheformer: a foundation model for single-cell and spatial omics

“…The diversity and complexity of these tasks are useful to thoroughly probe the model's performance and to evaluate the robustness of the learned representation and the model's ability to generalize to complex predictive tasks. Current results are promising but not entirely replicated in independent benchmarks [45][46][47][48][49][50] . Notably, to date, none of these models account for spatial relationships of cells during training, with the exception of CellPLM 40 , which, however, is trained on a limited dataset of 9 million dissociated and 2 million spatial transcriptomics cells 40 and not fine-tuned on spatial tasks beyond gene imputation.We propose Nicheformer, a novel spatial omics foundation model to understand tissue dependencies.…”