Evaluating Representations for Gene Ontology Terms

Abstract

Functions of proteins are annotated by Gene Ontology (GO) terms. As the amount of new sequences being collected is rising at a faster pace than the number of sequences being annotated with GO terms, there have been efforts to develop better annotation techniques. When annotating protein sequences with GO terms, one key auxiliary resource is the GO data itself. GO terms have definitions consisted of a few sentences describing some biological events, and are also arranged in a tree structure with specific terms being child nodes of generic terms. The definitions and positions of the GO terms on the GO tree can be used to construct vector representations for the GO terms. These GO vectors can then be integrated into existing prediction models to improve the classification accuracy. In this paper, we adapt the Bidirectional Encoder Representations from Transformers (BERT) to encode GO definitions into vectors. We evaluate BERT against the previous GO encoders in three tasks: (1) measuring similarity between GO terms (2) asserting relationship for orthologs and interacting proteins based on their GO annotations and (3) predicting GO terms for protein sequences. For task 3, we show that using GO vectors as additional prediction features increases the accuracy, primarily for GO terms with low occurrences in the manually annotated dataset. In all three tasks, BERT often outperforms the previous GO encoders.https://www.uniprot.org/

Annotating Gene Ontology terms for protein sequences with the Transformer model

“…To keep our software GOAT manageable for all users, we implement Transformer with only one head and 12 layers. We initialize the GO embedding E G as the pre-trained embedding BERT LAYER12 in [6] which transforms the GO definitions into vectors, where GOs having related definitions will have comparable vectors. Using pre-trained GO embeddings reduces the number of parameters in the Transformer, which can also reduce overfitting, use less GPU memory, and decrease run time.…”

“…We next evaluate whether our adaptation of Transformer can learn the co-occurrences of labels. Duong et al [6] noticed in their GO embedding that when one of the child-parent GO labels describes very broad biological events (e.g. low IC), then their vector representations may be far apart.…”

“…We suspect that Transformer can model co-occurrences of labels. To validate this, from the T-SNE plot of the vectors in the result section, we observe that a label and its ancestors will be nearby, even when their initial definition embeddings in [6] are dissimilar, as is the case when a term and its distant ancestors can have dissimilar definitions.…”

“…In this paper, we set the vectors represent the amino acids and GO labels to be in the same dimension; that is, E M and are vectors of the same size. To reduce the number of parameters, for the GO labels, we fix E G as the definition-based GO embeddings from [6] instead of setting it as a trainable parameter.…”

Automatic Gene Function Prediction in the 2020’s

“…Several such representations have been proposed, one of the first ones being clusDCA [41], which used random walks to learn features that reflect the GO graph topology. More recent approaches make use of advances in Natural Language Processing (NLP) to learn embeddings that reflect semantic meanings based on the term names and/or descriptions [42]. Theoretical work has shown the utility of embedding graph-structured data (as GO terms are) in hyperbolic rather than Euclidean spaces [43].…”