AWLCO: All-Window Length Co-Occurrence

The Complexity of the Co-Occurrence Problem

“…Theorem 1(a) and 1(b) together prove that our data structure has optimal space, and that with optimal space we cannot hope to support queries faster than O(log log n). In comparison to the state of the art by Sobel et al [15], Theorem 1(c) proves that we match 1 their solution, and also that the O( √ nq) space bound is tight to within polylogarithmic factors. We note that all of our results are based on simple and intuitive combinatorial ideas that simplify the state of the art.…”

“…Then lm(j) is the maximum dist(j, •)-value. As in [15], we maintain the dist(j, •)-values in a linked list that is dynamically reordered according to the well-known move-to-front rule. The algorithm works as follows.…”

“…Furthermore, it is similar to certain string problems, such as episode matching [4], where the goal is to determine all the substrings of S that occur a certain number of times within a given distance from each other. Whereas previous work is mostly concerned with identifying frequent patterns either in the whole string or in a sliding window of fixed length, Sobel, Bertram, Ding, Nargesian and Gildea [15] introduced the problem of studying a given pattern across all window lengths (i.e., determining co S,Q (i) for all i). They motivate the problem by listing potential applications such as training models for natural language processing (short and long co-occurrences of a set of words tend to represent respectively syntactic and semantic information), automatically organizing the memory of a computer program for good cache behaviour (variables that are used close to each other should be near each other in memory), and analyzing DNA sequences (co-occurrences of nucleotides in DNA provide insight into the evolution of viruses).…”

“…They motivate the problem by listing potential applications such as training models for natural language processing (short and long co-occurrences of a set of words tend to represent respectively syntactic and semantic information), automatically organizing the memory of a computer program for good cache behaviour (variables that are used close to each other should be near each other in memory), and analyzing DNA sequences (co-occurrences of nucleotides in DNA provide insight into the evolution of viruses). See [15] for a more detailed discussion of these applications.…”

“…Our work is inspired by [15]. They do not consider fast, individual queries, but instead they give an O( √ nq) space data structure from which they can determine co S,Q (i) for each i = 1, .…”