|
| |
|
|
oraclewiz
Posts: 39
Joined: 10/31/2003
Status: offline
 |
Robot.txt file - 2/28/2006 15:35:56
Alexa tells me that you should definitely have a robot.txt file in your root directory. They specify:
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. If its missing you may be not indexed at all.
The robots.txt file is a simple text file.(no HTML),
( User-agent: *
Disallow:)
that must be placed in your root directory, for example:
http://www.yourwebsite.com/robots.txt
Since I am newbe and file illierate, I'm not sure what this means. Is this my root directory file C:/MyWebs/oraclewiz
and does if have to be the first file in the hierarchy??
Using FP 2000 should I create another page and put this text in it and upload to GoDaddy my provider. And does if have to be the first file in the hierarchy??
|
|
|
|
Reflect
Posts: 4769
From: USA
Status: offline
|
RE: Robot.txt file - 2/28/2006 16:18:30
Hi,
Do not use a wysiwyg editor to make this file, use notepad.
Once created it goes in the root of your web site (where your index page resides). FP publish is OK to push it out with. As for hierarchy, don't worry about it.
Take care,
Brian
_____________________________
| |
|
|
|
coreybryant
Posts: 2422
Joined: 3/17/2002
From: Castle Rock CO USA
Status: offline
|
RE: Robot.txt file - 2/28/2006 17:44:20
Your root is basically where your index.html file is (your home page of your website). Good robots will follow them, bad robots will not.
It does not have to be the first file - chances are it will not be - but the name does have to be robots.txt
_____________________________
Corey R. Bryant
Merchant Accounts | Toll Free Numbers | My Blog | Expression Web Blog
| |
|
|
|
Mojo
Posts: 2431
From: Chicago
Status: offline
|
RE: Robot.txt file - 2/28/2006 18:53:56
quote:
If its missing you may be not indexed at all
Not true. Missing this file has zero impact on wether or not your site will be indexed.
_____________________________
Split Testing
Chicago Order Fulfillment
Emergency Kits
| |
|
|
|
Peppergal
Posts: 2204
Joined: 9/20/2002
Status: offline
|
RE: Robot.txt file - 3/23/2006 22:25:35
Here's a dumb question.
If I have a few pages in the main directory that I don't want indexed, and I just want to use the robots.txt file instead of <meta> tags on each of those pages, do I just use /page.html ? or do I have to have the entire URL?
or should I use the meta tags?
_____________________________
Northeast PA / Poconos/ Lake Wallenpaupack Real Estate
wallenpaupacklakeproperty.com
Karen's Real Estate Blog
| |
|
|
|
Reflect
Posts: 4769
From: USA
Status: offline
|
RE: Robot.txt file - 3/27/2006 10:56:31
Google sitemap explained (It's sort of new so don't sweat not knowing about it)....
http://www.google.com/webmasters/sitemaps/login
Take care,
Brian
_____________________________
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
. I then make sure to leave the page out of my google sitemap and my normal site map. 
I didn't know there was a difference! I've been away from frontpagewebmaster too long!! 