ACME: A scalable parallel system for extracting frequent patterns from a very long sequence

Abstract

Modern applications, including bioinformatics, time series, and web log analysis, require the extraction of frequent patterns, called motifs, from one very long (i.e., several gigabytes) sequence. Existing approaches are either heuristics that are error-prone, or exact (also called combinatorial) methods that are extremely slow, therefore, applicable only to very small sequences (i.e., in the order of megabytes). This paper presents ACME, a combinatorial approach that scales to gigabyte-long sequences and is the first to support supermaximal motifs. ACME is a versatile parallel system that can be deployed on desktop multi-core systems, or on thousands of CPUs in the cloud. However, merely using more compute nodes does not guarantee efficiency, because of the related overheads. To this end, ACME introduces an automatic tuning mechanism that suggests the appropriate number of CPUs to utilize, in order to meet the user constraints in terms of run time, while minimizing the financial cost of cloud resources. Our experiments show that, compared to the state of the art, ACME supports three orders of magnitude longer sequences (e.g., DNA for the entire human genome); handles large alphabets (e.g., English alphabet for Wikipedia); scales out to 16,384 CPUs on a supercomputer; and supports elastic deployment in the cloud.

Dynamic Configuration of Partitioning in Spark Applications

“…A natural choice to this end is to employ sampling, e.g., as in [13], [14]. However, sampling-based automated profile generation seems to be a particularly challenging task in Spark.…”

“…However, all these cost modeling and profiling techniques do not cover specific phenomena in Spark execution, such as super-linear speed-ups for small degrees of parallelism and performance degradation for large ones. The proposals in [13], [14] present a sampling-based approach to estimate the profile of a single embarrassingly parallel task, based on the behavior of some of its partitions. However, they assume that partitions are scheduled in multiple waves, whereas we have adopted a configuration, where all partitions are scheduled in a single wave but there are multiple interdependent tasks.…”

MASA: Motif-Aware State Assignment in Noisy Time Series Data

“…We first seek to find all maximal repeated subsequences of states in S ′ . A maximal subsequence is defined as a sequence that cannot be extended to either the left or right without changing the set of occurrences in S ′ [26]. We require that each repeated subsequence has at least L non-overlapping instances in S ′ .…”

“…Motif discovery is a common problem in time series data analysis [6]. Methods for finding motifs include random projection [4] and suffix arrays [26]. Some of these methods are event rather than numerically based and thus bypass the simultaneous problem of state assignment [23].…”

“…The problem of finding repeated patterns also appears in the field of computational genomics. ACME uses a combinatorial approach to find super-maximal motifs in DNA [26]. Other bioinformatics models use edit-distance approaches to find motifs that vary slightly in appearance over the course of the sequence [22].…”