I have a question about bots (Full Version)

All Forums >> [Web Development] >> General Web Development



Message

yb2 -> I have a question about bots (5/2/2006 12:50:42)

Not dirty bots or cheeky bots, but search engine bots[;)]

I want to know when a bot has/is crawling my sites without me crawling the logs, but I don't know how.

Can anyone point me in the right direction, please?



jaybee -> RE: I have a question about bots (5/2/2006 13:17:07)

Get a copy of AWStats if your host doesn't already offer it as an option. It lists robots as a separate line.

[image]local://upfiles/9075/7FD051C6B7744683A65BF16703741B4D.gif[/image]

quote:

It can analyze log files from all major server tools like Apache log files (NCSA combined/XLF/ELF log format or common/CLF log format), WebStar, IIS (W3C log format) and a lot of other web, proxy, wap, streaming servers, mail servers and some ftp servers.



yb2 -> RE: I have a question about bots (5/2/2006 13:24:32)

Thanks, it looks interesting. I was hoping to code some of this myself (eventually), any ideas on how I know it's a bot crawling the site as opposed to anyone else?



rdouglass -> RE: I have a question about bots (5/2/2006 13:32:34)

I read somewhere recently that a 'bot never returns a user agent request but I can't be positive. Let me see if I can find that article.



Reflect -> RE: I have a question about bots (5/2/2006 15:50:06)

Bots do return user agents, like googlebot as one example.

Here is a pretty updated list but there are PLENTY more out there...

http://www.jafsoft.com/searchengines/webbots.html

Take care,

Brian



yb2 -> RE: I have a question about bots (5/2/2006 16:08:10)

thanks all, this is all really helpful stuff.



Page: [1]

Valid CSS!




Forum Software © ASPPlayground.NET Advanced Edition 2.4.5 ANSI
0.0625