Efficient Visualization of Large-Scale Data Tables through Reordering and Entropy Minimization

Abstract

Abstract-Visualization of data tables with n examples and m columns using heatmaps provides a holistic view of the original data. As there are n! ways to order rows and m! ways to order columns, and data tables are typically ordered without regard to visual inspection, heatmaps of the original data tables often appear as noisy images. However, if rows and columns of a data table are ordered such that similar rows and similar columns are grouped together, a heatmap may provide a deep insight into the underlying data distribution. We propose an informationtheoretic approach to produce a well-ordered data table. In particular, we search for ordering that minimizes entropy of residuals of predictive coding applied on the ordered data table. This formalization leads to a novel ordering procedure, EMordering, that can be applied separately on rows and columns. For ordering of rows, EM-ordering repeats until convergence the steps of (1) rescaling columns and (2) solving a Traveling Salesman Problem (TSP) where rows are treated as cities. To allow fast ordering of large data tables, we propose an efficient TSP heuristic with modest O n log(n) time complexity. When compared to the existing state-of-the-art reordering approaches, we show that the method often provides heatmaps of higher visual quality, while being significantly more scalable. Moreover, analysis of real-world traffic and financial data sets using the proposed method, which allowed us to readily gain deeper insights about the data, further confirmed that EM-ordering can be a valuable tool for visual exploration of large-scale data sets.

A fast feature vector approach for revealing simplex and equi-correlation data patterns in reorderable matrices

“…The EM-ordering time complexity is O ( n log n ) . 11 The AVC time complexity, as implemented in Matrix Reordering Analyzer tool (MRA), is O ( n 3 ) . 14 The time complexity of TSP algorithm is not defined by Gurobi documentation.…”

“…Proximity matrix reordering algorithms are based on distinct approaches. Breadth-first search, depth-first search, reverse Cuthill–McKee, King’s algorithm, Degree, 7 visual assessment of (cluster) tendency, 8 OREO, 9 TSPCluster 10 and EM-ordering 11 are graph-based algorithms, which order graph nodes according to specific searching procedures (e.g. breadth-first search) or node characteristics (such as node degree or node similarity).…”

“…EM-ordering uses a specific TSP solver to discover matrix permutations that minimize entropy. 11 TSPCluster 10 also uses a TSP solver to solve the reordering problem for a given number of clusters.…”

Polar Sort: Combining Multidimensional Scaling and Polar Coordinates for Matrix Reordering

“…For the sake of completeness, we list the asymptotic time complexity of the reordering algorithms used at our experiments: Circumplex Sort -O(n²) (Rocha et al 2017); Classical MDS -O(n³) (Tzeng et al, 2008), EM-ordering -O(n log n) (Djuric & Vucetic, 2013), Polar Sort -O(n³), SMB+FV+Median -O(n²α(n²)) (Medina et al, 2016), Eigen Decomposition O(n³) (Dubrulle et al, 1971). We could not find a reliable reference with the asymptotic time complexity of SVD.…”