Efficient mixture model for clustering of sparse high dimensional binary data

Śmieja, Marek; Hajto, Krzysztof; Tabor, Jacek

doi:10.1007/s10618-019-00635-1

“…This can occur in pangenomics as the discovery rate of new families in the pangenome slightly decreases when new genomes are added. Mathematical solutions to this problem seem to exist [50][51][52] for example via the weighting of genomes (based on their respective contribution to the pangenome diversity) or via sparse partitioning methods. An improvement of NEM should include these solutions and could be a perspective of this work.…”

Section: Issues Resulting From High-dimensional Statistics and Parallmentioning

confidence: 99%

PPanGGOLiN: Depicting microbial diversity via a partitioned pangenome graph

Gautreau

¹

,

Bazin

²

,

Gachet

³

et al. 2020

View full text Add to dashboard Cite

The use of comparative genomics for functional, evolutionary, and epidemiological studies requires methods to classify gene families in terms of occurrence in a given species. These methods usually lack multivariate statistical models to infer the partitions and the optimal number of classes and don't account for genome organization. We introduce a graph structure to model pangenomes in which nodes represent gene families and edges represent genomic neighborhood. Our method, named PPanGGOLiN, partitions nodes using an Expectation-Maximization algorithm based on multivariate Bernoulli Mixture Model coupled with a Markov Random Field. This approach takes into account the topology of the graph and the presence/absence of genes in pangenomes to classify gene families into persistent, cloud, and one or several shell partitions. By analyzing the partitioned pangenome graphs of isolate genomes from 439 species and metagenome-assembled genomes from 78 species, we demonstrate that our method is effective in estimating the persistent genome. Interestingly, it shows that the shell genome is a key element to understand genome dynamics, presumably because it reflects how genes present at intermediate frequencies drive adaptation of species, and its proportion in genomes is independent of genome size. The graph-based approach proposed by PPanGGOLiN is useful to depict the overall genomic diversity of thousands of strains in a compact structure and provides an effective basis for very large scale comparative genomics. The software is freely available at https://github.com/labgem/ PPanGGOLiN. PLOS COMPUTATIONAL BIOLOGYPLOS Computational Biology | https://doi.org/10.

show abstract

“…Actually, it can be the case in pangenomics as the number of new families added to the pangenome slightly decreases when new genomes are added (see figure 3 in [1]). Mathematical solutions to this issue seem to exist [46,47,48] for example via the weighting of features, corresponding to the weighting of genomes in our case. An improved version of NEM should include this improvement and could be perspective of this work.…”

Section: Issues Resulting From High-dimensional Statisticsmentioning

confidence: 97%

PPanGGOLiN: depicting microbial diversity via a partitioned pangenome graph

Gautreau

¹

,

Bazin

²

,

Gachet

³

et al. 2019

Preprint

View full text Add to dashboard Cite

The use of comparative genomics for functional, evolutionary, and epidemiological studies requires methods to classify gene families in terms of occurrence in a given species. These methods usually lack multivariate statistical models to infer the partitions and the optimal number of classes and don't account for genome organization. We introduce a graph structure to model pangenomes in which nodes represent gene families and edges represent genomic neighborhood. Our method, named PPanGGOLiN, partitions nodes using an Expectation-Maximization algorithm based on multivariate Bernoulli Mixture Model coupled with a Markov Random Field. This approach takes into account the topology of the graph and the presence/absence of genes in pangenomes to classify gene families into persistent, cloud, and one or several shell partitions. By analyzing the partitioned pangenome graphs of isolate genomes from 439 species and metagenome-assembled genomes from 78 species, we demonstrate that our method is effective in estimating the persistent genome. Interestingly, it shows that the shell genome is a key element to understand genome dynamics, presumably because it reflects how genes present at intermediate frequencies drive adaptation of species, and its proportion in genomes is independent of genome size. The graph-based approach proposed by PPanGGOLiN is useful to depict the overall genomic diversity of thousands of strains in a compact structure and provides an effective basis for very large scale comparative genomics. The software is freely available at https://github.com/labgem/PPanGGOLiN.

show abstract

“…Generally, clustering can be divided into five categories: partitioning [ 10 , 11 ], hierarchical [ 12 , 13 ], model-based [ 14 , 15 ], density-based [ 16 , 17 , 18 ], and grid-based algorithms. Partitioned clustering is designed to discover clusters in the data by optimizing a given objective function.…”

Section: Introductionmentioning

confidence: 99%

Grid-Based Clustering Using Boundary Detection

Du

¹

,

Wu

²

2022

Entropy

View full text Add to dashboard Cite

Clustering can be divided into five categories: partitioning, hierarchical, model-based, density-based, and grid-based algorithms. Among them, grid-based clustering is highly efficient in handling spatial data. However, the traditional grid-based clustering algorithms still face many problems: (1) Parameter tuning: density thresholds are difficult to adjust; (2) Data challenge: clusters with overlapping regions and varying densities are not well handled. We propose a new grid-based clustering algorithm named GCBD that can solve the above problems. Firstly, the density estimation of nodes is defined using the standard grid structure. Secondly, GCBD uses an iterative boundary detection strategy to distinguish core nodes from boundary nodes. Finally, two clustering strategies are combined to group core nodes and assign boundary nodes. Experiments on 18 datasets demonstrate that the proposed algorithm outperforms 6 grid-based competitors.

show abstract

Efficient mixture model for clustering of sparse high dimensional binary data

Cited by 11 publications

References 70 publications

PPanGGOLiN: Depicting microbial diversity via a partitioned pangenome graph

PPanGGOLiN: Depicting microbial diversity via a partitioned pangenome graph

PPanGGOLiN: depicting microbial diversity via a partitioned pangenome graph

Grid-Based Clustering Using Boundary Detection

Contact Info

Product

Resources

About