Efficient crawling through URL ordering
WWW7 Proceedings of the seventh international conference on World Wide Web 7
Improvement of PageRank for Focused Crawler
SNPD '07 Proceedings of the Eighth ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing - Volume 02
FICA: A Fast Intelligent Crawling Algorithm
WI '07 Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence
Architecture for Parallel Crawling and Algorithm for Change Detection in Web Pages
ICIT '07 Proceedings of the 10th International Conference on Information Technology
A New Approach to Design Domain Specific Ontology Based Web Crawler
ICIT '07 Proceedings of the 10th International Conference on Information Technology
HAWK: A Focused Crawler with Content and Link Analysis
ICEBE '08 Proceedings of the 2008 IEEE International Conference on e-Business Engineering
URL Rule Based Focused Crawler
ICEBE '08 Proceedings of the 2008 IEEE International Conference on e-Business Engineering
| Hi-index | 0.00 |
Focused crawlers aim to search only the subset of the web related to a specific topic, and offer a potential solution to the problem. The major problem is how to retrieve the maximal set of relevant and quality pages. In this paper, We propose an architecture that concentrates more over page selection policy and page revisit policy The three-step algorithm for page refreshment serves the purpose. The first layer contributes to decision of page relevance using two methods. The second layer checks for whether the structure of a web page has been changed or not, the text content has been altered or whether an image is changed. Also a minor variation to the method of prioritizing URLs on the basis of forward link count has been discussed to accommodate the purpose of frequency of update. And finally, the third layer helps to update the URL repository.