D2S: Document-to-sentence framework for novelty detection

Authors:
Flora S. Tsai;Yi Zhang
Affiliations:
Nanyang Technological University, Singapore, Singapore;Nanyang Technological University, Singapore, Singapore
Venue:
Knowledge and Information Systems
Year:
2011

Citing 0
Cited 6

Multilingual novelty detection

Expert Systems with Applications: An International Journal
Design of an intelligent novelty detection application

International Journal of Innovative Computing and Applications
Dimensionality reduction for blog tag mining

International Journal of Web Engineering and Technology
Chinese categorization and novelty mining

PAKDD'11 Proceedings of the 15th Pacific-Asia conference on Advances in knowledge discovery and data mining - Volume Part II
A data-centric approach to feed search in blogs

International Journal of Web Engineering and Technology
Clustering web documents using hierarchical representation with multi-granularity

World Wide Web

Quantified Score

Hi-index	0.00

Visualization

Abstract

Novelty detection aims at identifying novel information from an incoming stream of documents. In this paper, we propose a new framework for document-level novelty detection using document-to-sentence (D2S) annotations and discuss the applicability of this method. D2S first segments a document into sentences, determines the novelty of each sentence, then computes the document-level novelty score based on a fixed threshold. Experimental results on APWSJ data show that D2S outperforms standard document-level novelty detection in terms of redundancy-precision (RP) and redundancy-recall (RR). We applied D2S on the document-level data from the TREC 2004 and TREC 2003 Novelty Track and find that D2S is useful in detecting novel information in data with a high percentage of novel documents. However, D2S shows a strong capability to detect redundant information regardless of the percentage of novel documents. D2S has been successfully integrated in a real-world novelty detection system.