|
| |
|
|
fade2black
Posts: 61
Joined: 5/11/2004
From: England, but now living in Southern Spain
Status: offline
 |
Quick robots.txt question - 6/1/2005 5:46:33
Hi
I have been reading up on robots.txt and then decided to set it up and upload to the server. I specified a couple of Disallows for directories I do not want picking up and that was it. Did this a week ago and not noticed any change. Probably too early to tell.
So, is that it? Surely there must be more to it than that. Do I need to add the Index="Follow"... and all the other robotic statements to the html? If so, I would be grateful for the statements and context to use.
To throw a spanner into the works, I checked a few No1 spot web sites on Google, which also use the robots.txt and found that some had the robot statements and some had none, nada, zilch...I am confused!
Thanks
Fade...
|
|
|
|
Reflect
Posts: 4769
From: USA
Status: offline
|
RE: Quick robots.txt question - 6/1/2005 9:39:51
I always use both being robots.txt and no index, no follow META. I even go farther with Google. The following will tell the bot not to archive. This will cause the cache link button on the SEEP to show the page as it is currently...
<meta name="Google" content="no archive, no follow">
or
<meta name="Google" content="noarchive">
depending on if I want the bot to crawl the links on that page. Full explanation can be found here...
http://www.google.com/webmasters/faq.html#cached
Then I also get specific with MSN's bot.
<meta name="msnbot" content="noindex, nofollow">
Same thing on that META you can exclude noindex or nofollow depending on how you want the page treated.
On the blank robots.txt question, garbage in garbage out. It, to me, is a reactionary measure taken by webmasters who do not understand. All they know, IMO, is the log files are showing a 404 for robots.txt so they just put one out there blank to resolve the perceived issue in the log file.
Take care,
Brian
_____________________________
| |
|
|
|
fade2black
Posts: 61
Joined: 5/11/2004
From: England, but now living in Southern Spain
Status: offline
|
RE: Quick robots.txt question - 6/1/2005 12:50:52
Thanks guys. Looks like I'm gonna be busy for a while.
Regarding your reply "On the blank robots.txt question". Not sure if there is some confusion here, but when I mentioned "some had the robot statements and some had none". I meant no robot statements in the HTML.
It appears that top ranking websites in Google (the ones that I checked anyway) that have the robots.txt don't always have any robot related statements in the HTML at all. Some do, some don't. This made me wonder if having these statements would improve ranking etc.
Much appreciated as always.....Fade
| |
|
|
|
BobbyDouglas
Posts: 5479
Joined: 5/15/2003
From: Arizona
Status: offline
|
RE: Quick robots.txt question - 6/1/2005 14:12:10
Why use meta tags if you are using a robots.txt?
_____________________________
Arizona Web Design - Mr Bobs Web Design in Arizona
The Arizona Web Hosting Challenge
| |
|
|
|
Reflect
Posts: 4769
From: USA
Status: offline
|
RE: Quick robots.txt question - 6/1/2005 16:23:01
For different outcomes. Look at the Google specific meta. You can control it to how you want it to crawl the site, directories, and pages.
I started doing this when I had a disallow on all bots in a certain directory. Much to my dismay I was "just checking" what pages in my site were listed and how in the SERPs. I noticed the dissallowed directory from googlebot showed pages in Googles SERPs. I then started researching and this is what I thought was the solution. I tested this on my next site and it kept all mentions of the page from Google.
Also keep in mind there are rouge bots out there. Some don't even check for a robots.txt. Anyway to combat these I will take, even banning via .htaccess if need be.
Sorry on my miss understanding on what you were saying about some sites having it or not. No it does not help for rankings. It only helps control what the bot for the SE sees....hopefully.
Take care,
Brian
_____________________________
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
