Web robot detection techniques: overview and limitations
Data Mining and Knowledge Discovery
| Hi-index | 0.00 |
It is a common practice to analyze an access log of an WWW server to obtain data, such as user agent vendor percentage, supplying information more suitably, or to measure information exposure rate (mainly for advertisement effectiveness). HTTP does have a 驴User-Agent驴 header to let user agents supply their product names for these kinds of use, but in fact relying solely on this tends to be inaccurate. This paper presents some considerations needed for log analysis to determine the user agent in a more precise way, and application to an actual access log resulted in 2 percent difference compared to conventional method.