Measuring Performance Metrics of Machine Learning Algorithms for Detecting and Classifying Transposable Elements

Abstract

Because of the promising results obtained by machine learning (ML) approaches in several fields, every day is more common, the utilization of ML to solve problems in bioinformatics. In genomics, a current issue is to detect and classify transposable elements (TEs) because of the tedious tasks involved in bioinformatics methods. Thus, ML was recently evaluated for TE datasets, demonstrating better results than bioinformatics applications. A crucial step for ML approaches is the selection of metrics that measure the realistic performance of algorithms. Each metric has specific characteristics and measures properties that may be different from the predicted results. Although the most commonly used way to compare measures is by using empirical analysis, a non-result-based methodology has been proposed, called measure invariance properties. These properties are calculated on the basis of whether a given measure changes its value under certain modifications in the confusion matrix, giving comparative parameters independent of the datasets. Measure invariance properties make metrics more or less informative, particularly on unbalanced, monomodal, or multimodal negative class datasets and for real or simulated datasets. Although several studies applied ML to detect and classify TEs, there are no works evaluating performance metrics in TE tasks. Here, we analyzed 26 different metrics utilized in binary, multiclass, and hierarchical classifications, through bibliographic sources, and their invariance properties. Then, we corroborated our findings utilizing freely available TE datasets and commonly used ML algorithms. Based on our analysis, the most suitable metrics for TE tasks must be stable, even using highly unbalanced datasets, multimodal negative class, and training datasets with errors or outliers. Based on these parameters, we conclude that the F1-score and the area under the precision-recall curve are the most informative metrics since they are calculated based on other metrics, providing insight into the development of an ML application.

InpactorDB: A Classified Lineage-Level Plant LTR Retrotransposon Reference Library for Free-Alignment Methods Based on Machine Learning

“…In the current (post) genomic era [ 86 , 87 ], there is a need for automating TE annotation [ 39 , 44 ] to quickly analyze the huge amount of genomic data. Machine learning algorithms have become popular in bioinformatics because they provide promising results in complex tasks and given the availability of large databases.…”

“…We used ML algorithms such as logistic regression (LR), linear discriminant analysis (LDA), K-nearest neighbors (KNN), multi-layer perceptron with one layer (MLP), random forest (RF), decision trees (DT), naïve Bayes network (NB), and support vector machine (SVM) to test the performance of the datasets. We used the F1-score as the performance metric, which is the harmonic mean of precision and sensitivity [ 39 ] and we used it as the accuracy indicator; we used k-mer frequencies with 1 ≤ k ≤ 6 as features, and we used scaling and dimensional reduction using principal component analysis (PCA) as pre-processing steps, according to [ 39 ].…”

“…As features, we used k-mer frequencies with 1 ≤ k ≤ 6. We also applied scaling and reduction of dimensionality using principal components analysis (PCA) as suggested by [ 39 ]. Each dataset was partitioned into 80% for training, 10% for validation, and 10% for testing.…”

“…Currently, a challenge in genomics is to reliably annotate TEs. These elements have certain characteristics that make their identification and classification a complex task [ 38 , 39 ], such as repetitiveness, structural and nucleotide diversity, complex mobilization dynamics (including nested insertions), and species specificity [ 18 , 40 , 41 ]. Although de novo, homology-based, structure-based, and comparative genomics bioinformatics methods (or a combination of several methods) can automatically detect and classify TEs [ 42 ] (for a review see [ 5 , 26 ]), all of these approaches have limitations due to the diversity of TE structures, the quality of genome assemblies into others, and the sole use of any of these cannot produce high quality results.…”

“…In addition, we removed the redundancy of elements through consensus creation following the same methodology of REPET, obtaining more than 67,000 sequences. This dataset constitutes a valuable resource for homology-based TE annotation, which is the most used approach [ 39 ], such as in RepeatMasker. Furthermore, this database also contributes to developing and testing ML-based algorithms for alignment-free and automatic annotation methods.…”