Contextual Pattern Mining and Counting

Given a string $P$ of length $m$, a longer string $T$ of length $n>m$, and two integers $l\geq 0$ and $r\geq 0$, the context of $P$ in $T$ is the set of all string pairs $(L,R)$, with $|L|=l$ and $|R|=r$, such that the string $LPR$ occurs in $T$. We introduce two problems related to the notion of context: (1) the Contextual Pattern Mining (CPM) problem, which given $T$, $(m,l,r)$, and an integer $\tau>0$, asks for outputting the context of each substring $P$ of length $m$ of $T$, provided that the size of the context of $P$ is at least $\tau$; and (2) the Contextual Pattern Counting (CPC) problem, which asks for preprocessing $T$ so that the size of the context of a given query string $P$ can be found efficiently. Both problems have direct applications in text mining and bioinformatics and are challenging to solve for realistically long strings. For CPM, we propose an efficient algorithm that either uses only internal memory, or a bounded amount of internal memory and external memory, which allows much larger datasets to be handled. For CPC, we propose a near-linear space index that can answer queries in near-optimal time. We further improve the practical performance of the CPC index by optimizations that exploit the LZ77 factorization of $T$ and an upper bound on the query length. Using billion-letter datasets from different domains, we show that the external memory version of our CPM algorithm can handle very large datasets using a small amount of internal memory while its runtime is comparable to that of the internal memory version. Interestingly, we also show that our optimized index for CPC outperforms an approach based on the state of the art for the reporting version of CPC in terms of query time, index size, construction time, and construction space, often by more than an order of magnitude.