View maintenance in a warehousing environment
SIGMOD '95 Proceedings of the 1995 ACM SIGMOD international conference on Management of data
Implementing data cubes efficiently
SIGMOD '96 Proceedings of the 1996 ACM SIGMOD international conference on Management of data
Using control charts for parameter estimation of a homogeneous Poisson process
CIE '96 Proceedings of the 19th international conference on Computers and industrial engineering
Towards a better understanding of Web resources and server responses for improved caching
WWW '99 Proceedings of the eighth international conference on World Wide Web
A scalable Web cache consistency architecture
Proceedings of the conference on Applications, technologies, architectures, and protocols for computer communication
On the scale and performance of cooperative Web proxy caching
Proceedings of the seventeenth ACM symposium on Operating systems principles
Synchronizing a database to improve freshness
SIGMOD '00 Proceedings of the 2000 ACM SIGMOD international conference on Management of data
Proceedings of the 9th international World Wide Web conference on Computer networks : the international journal of computer and telecommunications netowrking
An adaptive model for optimizing performance of an incremental web crawler
Proceedings of the 10th international conference on World Wide Web
Keeping Up with the Changing Web
Computer
The Evolution of the Web and Implications for an Incremental Crawler
VLDB '00 Proceedings of the 26th International Conference on Very Large Data Bases
World-wide web cache consistency
ATEC '96 Proceedings of the 1996 annual conference on USENIX Annual Technical Conference
World Wide Web caching: the application-level view of the Internet
IEEE Communications Magazine
Characterizing Web Document Change
WAIM '01 Proceedings of the Second International Conference on Advances in Web-Age Information Management
Effective page refresh policies for Web crawlers
ACM Transactions on Database Systems (TODS)
A framework for analysis of data freshness
Proceedings of the 2004 international workshop on Information quality in information systems
Modeling and Managing Content Changes in Text Databases
ICDE '05 Proceedings of the 21st International Conference on Data Engineering
Evolution of web site design patterns
ACM Transactions on Information Systems (TOIS)
Looking at both the present and the past to efficiently update replicas of web content
Proceedings of the 7th annual ACM international workshop on Web information and data management
Estimation of internet file-access/modification rates from indirect data
ACM Transactions on Modeling and Computer Simulation (TOMACS)
Adaptive pull-based policies for wide area data delivery
ACM Transactions on Database Systems (TODS)
Managing duplicates in a web archive
Proceedings of the 2006 ACM symposium on Applied computing
Modelling information persistence on the web
ICWE '06 Proceedings of the 6th international conference on Web engineering
A dataflow approach to efficient change detection of HTML/XML documents in WebVigiL
Computer Networks: The International Journal of Computer and Telecommunications Networking - Web dynamics
Stanford WebBase components and applications
ACM Transactions on Internet Technology (TOIT)
Efficient, automatic web resource harvesting
WIDM '06 Proceedings of the 8th annual ACM international workshop on Web information and data management
Temporal multi-page summarization
Web Intelligence and Agent Systems
Efficient Monitoring Algorithm for Fast News Alerts
IEEE Transactions on Knowledge and Data Engineering
Modeling and managing changes in text databases
ACM Transactions on Database Systems (TODS)
Weaving temporal and reliability aspects into a schema tapestry
Data & Knowledge Engineering
Longitudinal trends in academic web links
Journal of Information Science
Validating quicksand: Temporal schema versioning in τXSchema
Data & Knowledge Engineering
Recrawl scheduling based on information longevity
Proceedings of the 17th international conference on World Wide Web
Microscale evolution of web pages
Proceedings of the 17th international conference on World Wide Web
Geographic web usage estimation by monitoring DNS caches
Proceedings of the first international workshop on Location and the web
Maintaining dynamic channel profiles on the web
Proceedings of the VLDB Endowment
Characterization of the evolution of a news Web site
Journal of Systems and Software
Parallel crawler architecture and web page change detection
WSEAS Transactions on Computers
On the feasibility of geographically distributed web crawling
Proceedings of the 3rd international conference on Scalable information systems
Topical web crawling using weighted anchor text and web page change detection techniques
WSEAS Transactions on Information Science and Applications
Proceedings of the 3rd workshop on Information credibility on the web
Estimating the rate of web page updates
IJCAI'07 Proceedings of the 20th international joint conference on Artifical intelligence
Proceedings of the 18th ACM conference on Information and knowledge management
The global network of outdoor webcams: properties and applications
Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems
Quality of web search quantification, standardization and representation
IMSA '07 Proceedings of the Eleventh IASTED International Conference on Internet and Multimedia Systems and Applications
SHARC: framework for quality-conscious web archiving
Proceedings of the VLDB Endowment
Proceedings of the VLDB Endowment
The Online Market Observatory: A Domain Model Approach
KSEM '09 Proceedings of the 3rd International Conference on Knowledge Science, Engineering and Management
Foundations and Trends in Information Retrieval
Using visual pages analysis for optimizing web archiving
Proceedings of the 2010 EDBT/ICDT Workshops
Optimising context data dissemination and storage in distributed pervasive computing systems
Pervasive and Mobile Computing
The adaptive web
Coverage and timeliness analysis of search engines with webpage monitoring results
WISE'07 Proceedings of the 8th international conference on Web information systems engineering
Towards a quality-oriented real-time web crawler
WISM'10 Proceedings of the 2010 international conference on Web information systems and mining
The SHARC framework for data quality in Web archiving
The VLDB Journal — The International Journal on Very Large Data Bases
Time-weighted web authoritative ranking
Information Retrieval
Best-effort refresh strategies for content-based RSS feed aggregation
WISE'10 Proceedings of the 11th international conference on Web information systems engineering
Archiving the web using page changes patterns: a case study
Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries
Improving the quality of web archives through the importance of changes
DEXA'11 Proceedings of the 22nd international conference on Database and expert systems applications - Volume Part I
Coherence-oriented crawling and navigation using patterns for web archives
TPDL'11 Proceedings of the 15th international conference on Theory and practice of digital libraries: research and advanced technology for digital libraries
A cooperative model for wide area content delivery applications
OTM'05 Proceedings of the 2005 Confederated international conference on On the Move to Meaningful Internet Systems - Volume >Part I
State transfer graph: an efficient tool for webview maintenance
WAIM'05 Proceedings of the 6th international conference on Advances in Web-Age Information Management
Decomposition-Based optimization of reload strategies in the world wide web
WISE'05 Proceedings of the 6th international conference on Web Information Systems Engineering
Temporal shingling for version identification in web archives
ECIR'2010 Proceedings of the 32nd European conference on Advances in Information Retrieval
An ontology-guided approach to change detection of the semantic web data
Journal on Data Semantics V
mod_oai: an apache module for metadata harvesting
ECDL'05 Proceedings of the 9th European conference on Research and Advanced Technology for Digital Libraries
Learning the grammar of distant change in the world-wide web
AI'04 Proceedings of the 17th Australian joint conference on Advances in Artificial Intelligence
Schema-mediated exchange of temporal XML data
ER'06 Proceedings of the 25th international conference on Conceptual Modeling
As time goes by: discovering eras in evolving social networks
PAKDD'10 Proceedings of the 14th Pacific-Asia conference on Advances in Knowledge Discovery and Data Mining - Volume Part I
Adaptive change estimation in the context of online market monitoring
EUROCAST'11 Proceedings of the 13th international conference on Computer Aided Systems Theory - Volume Part I
Modelling web changes data recatched during A spread of internet virus
Mathematical and Computer Modelling: An International Journal
ICWE'12 Proceedings of the 12th international conference on Web Engineering
Sentimental Spidering: Leveraging Opinion Information in Focused Crawlers
ACM Transactions on Information Systems (TOIS)
Predicting content change on the web
Proceedings of the sixth ACM international conference on Web search and data mining
Temporal web dynamics and its application to information retrieval
Proceedings of the sixth ACM international conference on Web search and data mining
Hierarchical DHT-based name resolution for information-centric networks
Computer Communications
Reading the correct history?: modeling temporal intention in resource sharing
Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries
Extending sitemaps for ResourceSync
Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries
CUVIM: extracting fresh information from social network
WAIM'13 Proceedings of the 14th international conference on Web-Age Information Management
Wearable queries: adapting common retrieval needs to data and users
Proceedings of the 7th International Workshop on Ranking in Databases
Evolving networks: Eras and turning points
Intelligent Data Analysis - Dynamic Networks and Knowledge Discovery
| Hi-index | 0.00 |
Many online data sources are updated autonomously and independently. In this article, we make the case for estimating the change frequency of data to improve Web crawlers, Web caches and to help data mining. We first identify various scenarios, where different applications have different requirements on the accuracy of the estimated frequency. Then we develop several "frequency estimators" for the identified scenarios, showing analytically and experimentally how precise they are. In many cases, our proposed estimators predict change frequencies much more accurately and improve the effectiveness of applications. For example, a Web crawler could achieve 35% improvement in "freshness" simply by adopting our proposed estimator.