Structure-Aware Data Consolidation

Abstract

We present a structure-aware technique to consolidate noisy data, which we use as a pre-process for standard clustering and dimensionality reduction. Our technique is related to mean shift, but instead of seeking density modes, it reveals and consolidates continuous high density structures such as curves and surface sheets in the underlying data while ignoring noise and outliers. We provide a theoretical analysis under a Gaussian noise model, and show that our approach significantly improves the performance of many non-linear dimensionality reduction and clustering algorithms in challenging scenarios.

A comprehensive single cell transcriptional landscape of human hematopoietic progenitors

“…The structure-aware technique that we adopted in this paper work is aimed to at revealing and consolidating continuous, low-dimensional, and high-density structures in the underlying higher-dimensional data, whereas ignoring noise and outliers. The theory, proof of convergence to the exact underlying data manifolds (under Gaussian noise assumption) and an investigation of its performance under different scenario can be found in Wu et al 44 . Here we will briefly describe its discretized version formulation, i.e., representing densities by sets of sample points.…”

“…In general, given a value for radius size r , it returns an estimated optimal structure providing an accurate representation of data layout complexity and allowing for an interpretation in biological terms. In their paper 44 , authors suggest a method to help user in setting this critical parameter. Under the assumption of Gaussian distributed input data with known variance, this method estimates a lower bound for r able to guarantee convergence to the true m -dimensional manifold.…”

“…PBA algorithm is available at https://github.com/AllonKleinLab/PBA. Structure-aware data consolidation algorithm described in Wu, Shihao et al 44 has been implemented in R is available at https://github.com/BiascoLab/PrincipalDevelopementalTrajectories.…”

Point Set Registration for 3D Range Scans Using Fuzzy Cluster-Based Metric and Efficient Global Optimization

“…2(c.1), where one center (in the red ellipse) is placed in the noise area. For this issue, some filtering techniques can be applied to preprocess the noisy point sets, such as the structure-aware data consolidation [48] iteratively moving noisy outliers towards underlying, lower-dimensional structures. In this study, for simplicity, we use a different method to address this issue: The robust fuzzy c-means method in [41] is applied to postprocess the results of fuzzy c-means and drag the misplaced center(s) to the real point area; afterwards, a simple and noniterative pruning method is performed to remove noisy outliers.…”

“…2(c.2), where all the centers are placed in the real point area. Compared to the filtering method in [48], robust fuzzy c-means is a simpler implementation. In addition, a functioning depth camera is unlikely to give point…”