Use cases > Analyze the bot traffic

You may also be interested in only displaying statistics on web requests done by a specific bot. For example the frequency of the requests done by the Google bot will tell you how much the Google search engine likes your web site.

The application extracts the name of the bot from the user agent string in the Browser or BrowserFamily fields.

So for example to display only requests done by the Google bot you can use the following filter:

Browser='Googlebot 2.1'

However if you just do that everyone could send the user agent string of the Google bot and make you believe he's a real Google bot. To validate that this is really the Google bot you need to use the host name of the client IP address. A real Google bot will always come from an IP address with a host name with googlebot.com as domain.

By default host names are not determined from the IP address because it's slowing down the loading process (the program needs to query remote DNS servers). You need to enable the setting Resolve the host name from the IP address in the Log analysis settings. Then once you have loaded the log files you will see two new fields ClientHostName and ClientHostDomain. So to make sure you display only the Google bot traffic you just need to use the following filter:

Browser='Googlebot 2.1' AND ClientHostDomain='googlebot.com'

You could do exactly the same with the Bing bot with the following filter:

Browser='bingbot 2.0' AND ClientHostDomain='msn.com'