Which Model to Transfer? Finding the Needle in the Growing Haystack

Renggli, Cédric; Pinto, André Susano; Rimanic, Luka; Puigcerver, Joan; Ruiz, Carlos; Zhang, Ce; Lučić, Mario

doi:10.1109/cvpr52688.2022.00899

Cited by 6 publications

(16 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Figure 3 illustrates the dierent model search strategies along with their computational requirements. We remark that model search strategies were extensively studied in our work [37], whereas here we present an overview of these methods and facts that are important from the perspective of SHiFT. As in [37], we divide model search strategies into two main categories: (A) taskagnostic strategies, which are those that ignore the downstream dataset, and (B) task-aware strategies, those that do take the downstream dataset into consideration.…”

Section: Model Search Strategiesmentioning

confidence: 99%

“…Instead of ne-tuning the weights of the pre-trained network as described previously, one freezes them and only learns the weights of a newly initialized linear head. In a large empirical study we have shown that such a linear proxy can suer from a relatively high regret when trusting this search strategy over exhaustively ne-tuning all models and then picking the best one [37]. Nevertheless, this approach still represents one of the most powerful known search strategy.…”

Section: Model Search Strategiesmentioning

confidence: 99%

“…Proceedings of the VLDB Endowment, Vol. 16 1: Example search queries supported by SHiFT, returning an ordering of registered models by their... Q1: best reported upstream accuracy [14] Q2: best nearest neighbor classier accuracy [25,34] Q3: best linear classier accuracy [5,29,43] Q4: Q1 and Q3 (on models excluding the result of Q1) [37] Q5: highest (ne-tune) accuracy on the most similar task [1] rst pre-trained on typically large and possibly private upstream datasets, and are then made available via model repositories such as TF-Hub, PyTorch Hub, and HuggingFace. Given a new downstream dataset, representative for the ML task, a user picks some of these pre-trained models to ne-tune, which typically requires adding randomly initialized layers to parts of the pre-trained deep network, and tuning all the parameters using the limited amount of downstream data.…”

Section: Introductionmentioning

confidence: 99%

“…Simply downloading all these models can take hours, not to mention running inference of all these models over a given dataset and implementing state-of-the-art search strategies. As a consequence, many users in our experience simply resort to only using the latest model -this ignores the vast diversity of available models and, as we show in our previous systematic benchmark [37], can lead to a signicant quality gap.…”

Section: Introductionmentioning

confidence: 99%

“…The third challenge we observe is that model search is an active research area where new strategies are coming out quite frequently, yet easily implementing them in a system and comparing them to existing search strategies is a painful task. Just in the last three years, researchers proposed at least eight new strategies [1,5,9,14,25,29,34,37]. Having users to catch up with these latest developments can be tedious and potentially a huge waste of resources.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

SHiFT

et al. 2022

Self Cite

View full text Add to dashboard Cite

Transfer learning can be seen as a data- and compute-efficient alternative to training models from scratch. The emergence of rich model repositories, such as TensorFlow Hub, enables practitioners and researchers to unleash the potential of these models across a wide range of downstream tasks. As these repositories keep growing exponentially, efficiently selecting a good model for the task at hand becomes paramount. However, a single generic search strategy (e.g., taking the model with the highest linear classifier accuracy) does not lead to optimal model selection for diverse downstream tasks. In fact, using hybrid or mixed strategies can often be beneficial. Therefore, we propose SHiFT, the first downstream task-aware, flexible, and efficient model search engine for transfer learning. Users interface with SHiFT using the SHiFT-QL query language, which gives users the flexibility to customize their search criteria. We optimize SHiFT-QL queries using a cost-based decision maker and evaluate them on a wide rang of tasks. Motivated by the iterative nature of machine learning development, we further support efficient incremental executions of our queries, which requires a special implementation when jointly used with our optimizations.

show abstract

Section: Model Search Strategiesmentioning

confidence: 99%

Section: Model Search Strategiesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SHiFT

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

Deep learning-based proteomics enables accurate classification of bulk and single-cell samples

Krull,

Kühn,

Höhn

et al. 2024

Preprint

View full text Add to dashboard Cite

Proteins are the main drivers of cell function and disease, making their analysis a powerful technique to characterize determinants of cell identity and to identify biomarkers. Current proteomic technology has the breadth to profile thousands of proteins and even the sensitivity to access single cells, however limitations in throughput restrict its application, e.g. not allowing classification of samples according to biological or clinical status in large sample cohorts. Therefore, we developed a deep learning-based approach for the analysis of mass spectrometric (MS) data, assigning proteomic profiles to sample identity. Specifically, we designed an architecture referred to as Proformer, and show that it is superior to convolutional neural network-driven architectures, is explainable, and demonstrates robustness towards batch-effects. Based on its tabular approach, we highlight the integration of all four dimensions of proteomic measurements (retention time, mass-to-charge, intensity and ion mobility), and demonstrate enhanced sample discrimination involving a treatment with IFN-γ, despite its subtle effect on the cell's proteome. In addition, the Proformer is not restricted to proteomic depth, and can classify cells by cell type and their differentiation status even using single-cell proteomic data. Collectively, this work presents a novel deep learning-based model for rapid classification of proteomic data, with important future implications to enhance patient stratification, early detection and single-cell analysis.

show abstract