|
| |
|
|
yb2
Posts: 653
Joined: 1/30/2006
Status: offline
 |
I have a question about bots - 5/2/2006 12:50:42
Not dirty bots or cheeky bots, but search engine bots
I want to know when a bot has/is crawling my sites without me crawling the logs, but I don't know how.
Can anyone point me in the right direction, please?
_____________________________
it is natural for people not to see one's own faults, and to exaggerate other people's faults and failings.
Currently listening to: L'Enfer Des Formes by Stereolab
|
|
|
|
yb2
Posts: 653
Joined: 1/30/2006
Status: offline
|
RE: I have a question about bots - 5/2/2006 13:24:32
Thanks, it looks interesting. I was hoping to code some of this myself (eventually), any ideas on how I know it's a bot crawling the site as opposed to anyone else?
_____________________________
it is natural for people not to see one's own faults, and to exaggerate other people's faults and failings.
Currently listening to: L'Enfer Des Formes by Stereolab
| |
|
|
|
rdouglass
Posts: 9280
From: Biddeford, ME USA
Status: offline
|
RE: I have a question about bots - 5/2/2006 13:32:34
I read somewhere recently that a 'bot never returns a user agent request but I can't be positive. Let me see if I can find that article.
_____________________________
Don't take you're eye off your final destination.
ASP Checkbox Function Tutorial.
| |
|
|
|
Reflect
Posts: 4769
From: USA
Status: offline
|
RE: I have a question about bots - 5/2/2006 15:50:06
Bots do return user agents, like googlebot as one example.
Here is a pretty updated list but there are PLENTY more out there...
http://www.jafsoft.com/searchengines/webbots.html
Take care,
Brian
_____________________________
| |
|
|
|
yb2
Posts: 653
Joined: 1/30/2006
Status: offline
|
RE: I have a question about bots - 5/2/2006 16:08:10
thanks all, this is all really helpful stuff.
_____________________________
it is natural for people not to see one's own faults, and to exaggerate other people's faults and failings.
Currently listening to: L'Enfer Des Formes by Stereolab
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
