Scaling log-linear analysis to datasets with thousands of variables

Abstract

Association discovery is a fundamental data mining task. The primary statistical approach to association discovery between variables is log-linear analysis. Classical approaches to log-linear analysis do not scale beyond about ten variables. We have recently shown that, if we ensure that the graph supporting the log-linear model is chordal, log-linear analysis can be applied to datasets with hundreds of variables without sacrificing the statistical soundness [21]. However, further scalability remained limited, because state-of-the-art techniques have to examine every edge at every step of the search. This paper makes the following contributions: 1) we prove that only a very small subset of edges has to be considered at each step of the search; 2) we demonstrate how to efficiently find this subset of edges and 3) we show how to efficiently keep track of the best edges to be subsequently added to the initial model. Our experiments, carried out on real datasets with up to 2000 variables, show that our contributions make it possible to gain about 4 orders of magnitude, making log-linear analysis of datasets with thousands of variables possible in seconds instead of days.

A Statistically Efficient and Scalable Method for Exploratory Analysis of High-Dimensional Data

“…We carry out further experiments on another real dataset: "Finance stock performance of the companies" used in [34], which contains 20 years financial performance of 490 companies. The number of samples in the dataset is 3450, where the financial footprints in individual days are considered as samples.…”

“…the number of variables describing each sampled data point. For example, one may be interested in recovering the gene interaction network over large number of genes based on only a handful of collected gene expression samples [33], or recovering the associations of stock performance in a set of companies based on large number of samples [34]. These problem scenarios make association discovery challenging due to different reasons.…”

Inferring Two-Level Hierarchical Gaussian Graphical Models to Discover Shared and Context-Specific Conditional Dependencies from High-Dimensional Heterogeneous Data

“…Hence, it is not required to check the candidature and to compute the scoring function of all candidate edges at every step. According to [32], the addition of an edge (a, b) to the candidate model affects the minimal separators between following node pairs: (a) a and the neighbors of b, (b) b and the neighbors of a, and (c) neighbors of a and b. So that we only recompute the candidature and scoring function of above-mentioned node pairs (i.e., edges) which takes O(|V|) times.…”

Penggalian Informasi Menggunakan Model Terdekomposisi: Aplikasi Pada Riset Penemuan Antibiotik

“…Prioritized Chordalysis [9] mengetahui bahwa untuk garis (a, b) yang sama, skornya juga sama, kecuali separator minimal Sab berbeda. Dengan demikian, pada suatu iterasi, suatu garis butuh dihitung kembali skornya apabila separator minimalnya berubah.…”