Efficient crawling through URL ordering
WWW7 Proceedings of the seventh international conference on World Wide Web 7
Breadth-first crawling yields high-quality pages
Proceedings of the 10th international conference on World Wide Web
Mercator: A scalable, extensible Web crawler
World Wide Web
Characterizing Web Document Change
WAIM '01 Proceedings of the Second International Conference on Advances in Web-Age Information Management
Design and Implementation of a High-Performance Distributed Web Crawler
ICDE '02 Proceedings of the 18th International Conference on Data Engineering
Surfing Notes: An Integrated Web Annotation and Archiving Tool
WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 03
| Hi-index | 0.00 |
CRAYSE1 is a SEarch WHIle CRAwl application, intended to perform fast searching of text in web pages. A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner. This process is also called spidering. Search engines, use spidering as a means of providing up-to-date data. Most of the existing web-crawlers archive the contents of the web starting from the input URL. Search engines index the results of web-crawlers and then perform searching when queried. As such, the searching is not performed while crawling. Hence such softwares can not be used for general use by web browsers. Also, the existing search mechanism in web browsers, search only on the current page and not recursively through all the links present in that page. In order to overcome such disadvantages, we propose in this paper to implement a web crawler that searches for a pattern efficiently and recursively through all the links including pdf links while crawling. CRAYSE can be used as a general purpose open source software by web browsers. It can also be used for offine searching. Further, the applications that require selective archival of web pages (based on the presence of a key word), can deploy CRAYSE for efficient search operations. This paper focusses on the design and implementation of CRAYSE and its demonstration through web applications.