Optimal transport, mean partition, and uncertainty assessment in cluster analysis

Li, Jia; Seo, Beomseok; Lin, Lin

doi:10.1002/sam.11418

Cited by 17 publications

(20 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, they share a common challenge, the lack of a gold standard to assess quality and performance, and the lack of methods for the selection of the number of groups (modules/clusters). The notion of stability has been used extensively in the clustering literature as a surrogate for performance [4, 12–14, 20, 22, 26, 38]. It is therefore natural to bridge stability estimation for clustering to the module detection problem in graphs.…”

Section: Discussionmentioning

confidence: 99%

“…In lieu of a gold standard, different forms of cluster stability have been used as a surrogate to assess performance. Stability estimates capture how stable the clusterings are over several different representations of the data, which are derived through subsetting, cross‐validation, data noising or re‐sampling, among others [4, 12, 14, 15, 20, 22, 26, 36–38].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A framework for stability‐based module detection in correlation graphs

Tian

Blair

et al. 2021

Statistical Analysis

View full text Add to dashboard Cite

Graphs can be used to represent the direct and indirect relationships between variables, and elucidate complex relationships and interdependencies. Detecting structure within a graph is a challenging problem. This problem is studied over a range of fields and is sometimes termed community detection, module detection, or graph partitioning. A popular class of algorithms for module detection relies on optimizing a function of modularity to identify the structure. In practice, graphs are often learned from the data, and thus prone to uncertainty. In these settings, the uncertainty of the network structure can become exaggerated by giving unreliable estimates of the module structure. In this work, we begin to address this challenge through the use of a nonparametric bootstrap approach to assessing the stability of module detection in a graph. Estimates of stability are presented at the level of the individual node, the inferred modules, and as an overall measure of performance for module detection in a given graph. Furthermore, bootstrap stability estimates are derived for complexity parameter selection that ultimately defines a graph from data in a way that optimizes stability. This approach is utilized in connection with correlation graphs but is generalizable to other graphs that are defined through the use of dissimilarity measures. We demonstrate our approach using a broad range of simulations and on a metabolomics dataset from the Beijing Olympics Air Pollution study. These approaches are implemented using bootcluster package that is available in the R programming language.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A framework for stability‐based module detection in correlation graphs

Tian

Blair

et al. 2021

Statistical Analysis

View full text Add to dashboard Cite

show abstract

“…The iris data was examined using the OTclust package (Li et al, 2019; Zhang et al, 2020). Figure 4b shows the stability for k ‐means as a measure of overall tightness across a range of

k

values.…”

Section: Approaches To Clustering Stabilitymentioning

confidence: 99%

Stability estimation for unsupervised clustering: A review

Liu

Blair

2022

WIREs Computational Stats

View full text Add to dashboard Cite

Cluster analysis remains one of the most challenging yet fundamental tasks in unsupervised learning. This is due in part to the fact that there are no labels or gold standards by which performance can be measured. Moreover, the wide range of clustering methods available is governed by different objective functions, different parameters, and dissimilarity measures. The purpose of clustering is versatile, often playing critical roles in the early stages of exploratory data analysis and as an endpoint for knowledge and discovery. Thus, understanding the quality of a clustering is of critical importance. The concept of stability has emerged as a strategy for assessing the performance and reproducibility of data clustering. The key idea is to produce perturbed data sets that are very close to the original, and cluster them. If the clustering is stable, then the clusters from the original data will be preserved in the perturbed data clustering. The nature of the perturbation, and the methods for quantifying similarity between clusterings, are nontrivial, and ultimately what distinguishes many of the stability estimation methods apart. In this review, we provide an overview of the very active research area of cluster stability estimation and discuss some of the open questions and challenges that remain in the field. This article is categorized under:Statistical Learning and Exploratory Methods of the Data Sciences > Clustering and Classification

show abstract

“…For example, in unsupervised clustering, cluster labels are named arbitrarily only as symbols to distinguish groups. Since clusters generated in multiple results usually do not correspond to each other sharply, OT is used to match clusters in different results [25]. It is sometimes improper to assume that the clustering results are random realizations of one underlying "truth".…”

Section: Optimal Transport With Relaxed Marginal Constraintsmentioning

confidence: 99%

“…Wasserstein distance has also been used for robust supervised learning [22]- [24]. In the case of unsupervised learning, OT readily applies to the issue of aligning clustering results (the consistent cluster labeling issue), which then forms the basis for ensemble clustering and uncertainty analysis for clustering [25], [26].…”

Section: Introductionmentioning

confidence: 99%

Optimal Transport With Relaxed Marginal Constraints

Lin

2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

Optimal transport (OT) is a principled approach for matching, having achieved success in diverse applications such as tracking and cluster alignment. It is also the core computation problem for solving the Wasserstein metric between probabilistic distributions, which has been increasingly used in machine learning. Despite its popularity, the marginal constraints of OT impose fundamental limitations. For some matching or pattern extraction problems, the framework of OT is not suitable, and post-processing of the OT solution is often unsatisfactory. In this paper, we extend OT by a new optimization formulation called Optimal Transport with Relaxed Marginal Constraints (OT-RMC). Specifically, we relax the marginal constraints by introducing a penalty on the deviation from the constraints. Connections with the standard OT are revealed both theoretically and experimentally. We demonstrate how OT-RMC can easily adapt to various tasks by three highly different applications in image analysis and single-cell data analysis. Quantitative comparisons have been made with OT and another commonly used matching scheme to show the remarkable advantages of OT-RMC.INDEX TERMS Optimal transport, linear programming, pattern extraction, fixed target matching, singlecell data analysis, cluster alignment

show abstract

Optimal transport, mean partition, and uncertainty assessment in cluster analysis

Cited by 17 publications

References 31 publications

A framework for stability‐based module detection in correlation graphs

A framework for stability‐based module detection in correlation graphs

Stability estimation for unsupervised clustering: A review

Optimal Transport With Relaxed Marginal Constraints

Contact Info

Product

Resources

About