Board Forum Crawling: A Web Crawling Method for Web Forum

Authors:
Yan Guo;Kui Li;Kai Zhang;Gang Zhang
Affiliations:
Software Division, ICT, CAS;Software Division, ICT, CAS;Software Division, ICT, CAS;Software Division, ICT, CAS
Venue:
WI '06 Proceedings of the 2006 IEEE/WIC/ACM International Conference on Web Intelligence
Year:
2006

Citing 0
Cited 9

iRobot: an intelligent crawler for web forums

Proceedings of the 17th international conference on World Wide Web
Exploring traversal strategy for web forum crawling

Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Covertly probing underground economy marketplaces

DIMVA'10 Proceedings of the 7th international conference on Detection of intrusions and malware, and vulnerability assessment
FoCUS: learning to crawl web forums

Proceedings of the 21st international conference companion on World Wide Web
Intelligent crawling of web applications for web archiving

Proceedings of the 21st international conference companion on World Wide Web
Complete-Thread extraction from web forums

APWeb'12 Proceedings of the 14th Asia-Pacific international conference on Web Technologies and Applications
Automatically extracting user reviews from forum sites

Computers & Mathematics with Applications
A Generalized Links and Text Properties Based Forum Crawler

WI-IAT '12 Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology - Volume 01
Intelligent and adaptive crawling of web applications for web archiving

ICWE'13 Proceedings of the 13th international conference on Web Engineering

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a new method of Board Forum Crawling to crawl Web forum. This method exploits the organized characteristics of the Web forum sites and simulates human behavior of visiting Web Forums. The method starts crawling from the homepage, and then enters each board of the site, and then crawls all the posts of the site directly. Board Forum Crawling can crawl most meaningful information of a Web forum site efficiently and simply. We experimentally evaluated the effectiveness of the method on real Web forum sites by comparing with the traditional breadth-first crawling. We also used this method in a real project, and 12000 Web forum sites have been crawled successfully. These results show the effectiveness of our method.