|
| |
|
|
_gail
Posts: 2876
From: So FL
Status: offline
 |
robots: nonindex - 4/4/2002 12:25:22
I added the "robots: nonindex" variables to the pages of a website yet when I did a search of the site in a few search engines, almost all links to the website were pulled up.
Am I using the correct name and value? Any suggestions as to how to get pages to the site not to appear in search engines?
Gail
|
|
|
|
pageoneresults
Posts: 1001
From: Orange, CA USA
Status: offline
|
RE: robots: nonindex - 4/4/2002 13:10:01
Hi Gail! Actually you should be using none which is equivalent to noindex, nofollow. Here is a detailed explanation of the META Robots Tag. I can tell you from experience that few Search Engines pay attention to this META tag and the best way to prevent spidering is to create a robots.txt file instead. Even then, not all robots obey the directives. Best thing to do is not have the pages online or put them into a password protected directory.
This is a tip that applies to those who do not have control over placing a robots.txt file in their root directory.
This tag is meant to provide users who cannot control the /robots.txt file at their sites. It provides a last chance to keep their content out of search services. It was decided not to add syntax to allow robot specific permissions within the robots meta tag (as is possible in the /robots.txt file).
<meta name="robots" content="robots-terms">
The content robot-terms is a comma separated list that may contain the following keywords (without regard to case): all, none, index, noindex, follow and nofollow.
none
Tells all robots to ignore this page (equivalent to: noindex, nofollow).
all
There are no restrictions on indexing this page, or following links from this page to determine pages to index (equivalent to: index, follow).
index
All robots are welcome to include this page in search services.
noindex
This page may not be indexed by a search service.
follow
Robots are welcome to follow links from this page to find other pages.
nofollow
Robots are not to follow links from this page.
If this meta tag is missing, or if there is no content, or the robot terms are not specified, then the robot terms will be assumed to be "index, follow" (e.g. "all"). If the keyword all is found in the robots terms list it overrides all other values. That is, a robots terms that is "nofollow, all, noindex, nofollow", would effectively be "all". If the robots terms contains contradictory information (e.g. "follow, nofollow, follow") then the robot is free to do whatever it wishes with regard to the behavior being addressed (in this case the follow behavior).
A robot terms consisting only of noindex allows the subsidiary links to be explored, even though the page is not to be indexed. A robots terms consisting only of nofollow allows the page to be indexed, but no links from the page are explored (this may be useful if the page is a free entry point into pay per view content.
If anyone has any questions related to this topic, please feel to contact pageoneresults and I'll do my best to provide answers.
In regards to the robots meta tag, the index and follow are pretty much useless. If your site is void of these tags, it means the same thing. Don't clutter your metas with information that does not need to be there.
SEO Consultants Directory
Search Engine Optimization Tips
Edited by - pageoneresults on 04/04/2002 13:15:42
| |
|
|
|
_gail
Posts: 2876
From: So FL
Status: offline
|
RE: robots: nonindex - 4/4/2002 17:58:06
As always, thanks pageoneresults. But some of that stuff is a bit overwhelming (ie. robots.txt file).
How much of this can be done via the page properties > custom menus directly with FrontPage?
Gail
Edited by - Gail on 04/04/2002 18:00:27
| |
|
|
|
pageoneresults
Posts: 1001
From: Orange, CA USA
Status: offline
|
RE: robots: nonindex - 4/4/2002 18:09:18
I would say none of it unfortunately. When it comes to advanced stuff like this, FP is not set up to handle it. This is information that you'll need to hand code and put into place. I'd be happy to help you set up a robots.txt file and show you where to put it, that is a simple one.
I don't give much weight to the Robots META Tag as I've tested it in the past and some robots obeyed, others did not. Even with the robots.txt file, some obey the directives and others do not.
SEO Consultants Directory
Search Engine Optimization Tips
Edited by - pageoneresults on 04/04/2002 18:16:31
| |
|
|
|
pageoneresults
Posts: 1001
From: Orange, CA USA
Status: offline
|
RE: robots: nonindex - 4/4/2002 18:44:21
Okay, here is what I need...
1. The URL of the site.
2. A list of all URL's that you do not want indexed. Please provide me with the Absolute URL for example...
http //www yoursite.com/sub-directory/page-name.htm.
3. Do you have FTP access to your site? If so, you will need to upload the robots.txt file I send you into the Root Directory of your web. It won't work in any other directory but the root.
4. If you don't have access, I believe you can upload the robots.txt file I send you using the publish feature of FP. Someone please correct me here if I'm wrong. The thing we need to make sure of is that the file retains its original formatting and does not end up with Mac or DOS Line Enders (another topic in itself).
I'll await your reply. You can send mail to gail@123seo.com, thanks!
P.S. Keep in mind, that anyone can view your robots.txt file so if you are planning on hiding anything from the general public, this is not the way to do it. This is strictly to keep the spiders from indexing the content of those pages and nothing else.
SEO Consultants Directory
Search Engine Optimization Tips
Edited by - pageoneresults on 04/04/2002 18:45:53
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
) by studying what you provide because right now I haven't the foggiest. 
. 