|
Starhugger -> Weird 404 errors - robots?? (5/14/2006 1:00:56)
|
I have been seeing some weird 404 requests in my stats, especially in the last couple of months. I get hundreds of partial webpage names or weird compound names. For example, the accurate "folder/webpage.htm" might show as "folder/webpa" or "folder/webpage1.htm/webpage2.htm".
I'm assuming these are robots "testing" for something in my site. I believe Inktomi does this, although I don't know what it is they get out of doing this. But I doubt this stuff is Inktomi because I've seen Inktomi (or what I assume is them) for a long time, and this is different and seems to have just started in the last few months.
It's driving me nuts because there's like 11 screens of the stuff so far this month!! I can't see if I have any real 404s because all this garbage is in the way.
Does anyone know which robots might be doing this? Are these guys legitimate or are they up to no good? Does anyone know what they're looking for when they do this? Are these hack attempts??
If I can narrow it down to particular robot(s) then I can probably block them through robots.txt or .htaccess. But if they're legitimate and I would lose traffic by blocking them, then I guess I'll have to live with them. Sigh... [8|]
Thanks for any information anyone can give me about this.
Starhugger
|
|
|
|
