Algorithms on strings, trees, and sequences: computer science and computational biology
Algorithms on strings, trees, and sequences: computer science and computational biology
Genetic Programming Prediction of Stock Prices
Computational Economics
Modern Cryptography, Probabilistic Proofs, and Pseudorandomness
Modern Cryptography, Probabilistic Proofs, and Pseudorandomness
IEEE Transactions on Knowledge and Data Engineering
Assessing the Statistical Significance of Overrepresented Oligonucleotides
WABI '01 Proceedings of the First International Workshop on Algorithms in Bioinformatics
Identification of Non-Random Patterns in Structural and Mutational Data: the Case of Prion Protein
CSB '03 Proceedings of the IEEE Computer Society Conference on Bioinformatics
Detection of Significant Sets of Episodes in Event Sequences
ICDM '04 Proceedings of the Fourth IEEE International Conference on Data Mining
Reliable detection of episodes in event sequences
Knowledge and Information Systems
Handbook of Mathematical Functions, With Formulas, Graphs, and Mathematical Tables,
Handbook of Mathematical Functions, With Formulas, Graphs, and Mathematical Tables,
Journal of Computing Sciences in Colleges - Papers of the Fourteenth Annual CCSC Midwestern Conference and Papers of the Sixteenth Annual CCSC Rocky Mountain Conference
Probability: Theory and Examples
Probability: Theory and Examples
Most significant substring mining based on chi-square measure
PAKDD'10 Proceedings of the 14th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part I
Probabilistic techniques for intrusion detection based on computer audit data
IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans
| Hi-index | 0.00 |
The problem of identification of statistically significant patterns in a sequence of data has been applied to many domains such as intrusion detection systems, financial models, web-click records, automated monitoring systems, computational biology, cryptology, and text analysis. An observed pattern of events is deemed to be statistically significant if it is unlikely to have occurred due to randomness or chance alone. We use the chi-square statistic as a quantitative measure of statistical significance. Given a string of characters generated from a memoryless Bernoulli model, the problem is to identify the substring for which the empirical distribution of single letters deviates the most from the distribution expected from the generative Bernoulli model. This deviation is captured using the chi-square measure. The most significant substring (MSS) of a string is thus defined as the substring having the highest chi-square value. Till date, to the best of our knowledge, there does not exist any algorithm to find the MSS in better than O(n2) time, where n denotes the length of the string. In this paper, we propose an algorithm to find the most significant substring, whose running time is O(n3/2) with high probability. We also study some variants of this problem such as finding the top-t set, finding all substrings having chi-square greater than a fixed threshold and finding the MSS among substrings greater than a given length. We experimentally demonstrate the asymptotic behavior of the MSS on varying the string size and alphabet size. We also describe some applications of our algorithm on cryptology and real world data from finance and sports. Finally, we compare our technique with the existing heuristics for finding the MSS.