Web objectionable text content detection using topic modeling technique
Expert Systems with Applications: An International Journal
| Hi-index | 0.00 |
Hidden Markov model (HMM) is widely used in time series modeling. Usually, it is necessarily to calculate the sequence’s likelihood w.r.t. HMM to evaluate the similarity between the sequence and the HMM. Hence, it is required to provide a method to select a best threshold value that can determine whether the sequence is well approximated by the model or not. However, this process is usually done manually. Here, we provide a method (HTDM) to determine the threshold automatically. Based on likelihood statistic, we conclude that the likelihood is subjected to normal distribution, and then standard deviation of the distribution is estimated. Hence, the distance threshold value can be achieved based on the rule of “three sigma”. In the experiment, we make performance comparison between the HMM-based hierarchical clustering algorithm HHCH using HTDM, and algorithm HBHCTS in which threshold is set by manual. Experiment results show that the proposed method is effective on both syntax dataset and real world dataset.