TopoBERT: Exploring the topology of fine-tuned word representations

Abstract

Transformer-based language models such as BERT and its variants have found widespread use in natural language processing (NLP). A common way of using these models is to fine-tune them to improve their performance on a specific task. However, it is currently unclear how the fine-tuning process affects the underlying structure of the word embeddings from these models. We present TopoBERT, a visual analytics system for interactively exploring the fine-tuning process of various transformer-based models – across multiple fine-tuning batch updates, subsequent layers of the model, and different NLP tasks – from a topological perspective. The system uses the mapper algorithm from topological data analysis (TDA) to generate a graph that approximates the shape of a model’s embedding space for an input dataset. TopoBERT enables its users (e.g. experts in NLP and linguistics) to (1) interactively explore the fine-tuning process across different model-task pairs, (2) visualize the shape of embedding spaces at multiple scales and layers, and (3) connect linguistic and contextual information about the input dataset with the topology of the embedding space. Using TopoBERT, we provide various use cases to exemplify its applications in exploring fine-tuned word embeddings. We further demonstrate the utility of TopoBERT, which enables users to generate insights about the fine-tuning process and provides support for empirical validation of these insights.

Topological data analysis reveals core heteroblastic and ontogenetic programs embedded in leaves of grapevine (Vitaceae) and maracuyá (Passifloraceae)

“…As an alternate approach, we turned to the Mapper algorithm, a method that has successfully identified novel relationships within other high-dimensional datasets [19,29,31]. A key advantage of Mapper is its ability to interrogate datasets using several different lenses, revealing structure and relationships from specific, mathematically-defined perspectives that are PLOS COMPUTATIONAL BIOLOGY hidden from traditional approaches (see Materials and Methods).…”

Topological data analysis reveals a core heteroblastic program embedded in leaves of grapevine and maracuyá

“…As an alternate approach, we turned to the Mapper algorithm, a method that has successfully identified novel relationships within other high-dimensional datasets (Palande et al 2022;Rathore et al 2023;Rizvi et al 2017). A key advantage of Mapper is its ability to interrogate datasets using several different lenses, revealing structure and relationships from specific, mathematicallydefined perspectives that are hidden from traditional approaches (see Materials and Methods).…”