Exluding bots from reports

Posted By: bern ()
Posted On: 2006-Jan-05 13:12

So its been years since I have been closely involved with web analytics. The last time I did so I was using cookie tracking with webtrends. I want to say that I used to filter out robot and spider IP adresses from the reporting as well.

I am now using indextools - is the practice of filtering spiders still in practice? Does doing so interfere with search engine & keyword referral reports?

Thanks



Posted By: OAC (Staff)
Posted On: 2006-Jan-10 00:51

"is the practice of filtering spiders still in practice?"

Yes

"Does doing so interfere with search engine & keyword referral reports?"

No. The log analyzers distinguish between refewrrals from search engines and the rest of the entries.

So why filter spiders from the log file analysis? I don't filter "ggod spiders" eg Googlebot as I want to know that the spiders from the major search engines are active. However, I filter nuisance spiders from the analysis because they distort the page views and hits numbers.

Alternatively, as some webmasters do, you could actually ban the IPs of the sources of such nuisance spiders, via the .htaccess file. Then your logs aren't polluted by their visits.