|
| |
|
|
Peppergal
Posts: 2207
Joined: 9/20/2002
Status: offline
 |
Robots.txt or Robots META tag? - 9/21/2003 22:03:13
Is using a Robots meta tag a reasonable alternative to making up a robots.txt page?
What are the advantages in disallowing a robot to crawl a page, (for example, a "contact us" page)? I really have nothing that I want to keep the robots from crawling (or at least, i don't think i do) as most of the pages on the sites I design, with the one exception of the "contact us" page, are all relevant to the keywords people would be using to find the site to begin with.
Or am I totally misunderstanding this?
If I do make a robots.txt page, I have to make it in Notepad and upload it via FTP or thru the upload feature my webhost provides, right?
_____________________________
Northeast PA / Poconos/ Lake Wallenpaupack Real Estate
wallenpaupacklakeproperty.com
Karen's Real Estate Blog
|
|
|
|
fiachra
Posts: 338
Joined: 5/12/2002
From:
Status: offline
|
RE: Robots.txt or Robots META tag? - 9/22/2003 16:00:55
quote:
ORIGINAL: Peppergal
1. Is using a Robots meta tag a reasonable alternative to making up a robots.txt page?
2. What are the advantages in disallowing a robot to crawl a page, (for example, a "contact us" page)? I really have nothing that I want to keep the robots from crawling (or at least, i don't think i do) as most of the pages on the sites I design, with the one exception of the "contact us" page, are all relevant to the keywords people would be using to find the site to begin with.
3. Or am I totally misunderstanding this?
4. If I do make a robots.txt page, I have to make it in Notepad and upload it via FTP or thru the upload feature my webhost provides, right?
1. No, many bots ignore that metatag especially the email spam harvester bots. It's no harm to have it in there though. Well behaved bots do take heed of it.
2. In your case no real disadvantages with the exception of the spam harvester bots.
Note that spam harvester bots tend to ignore robots.txt anyway. So if you don't want to be bothered with it don't bother with it. It's different in each case.
3. Nope you're not. Just that the two are a little different.
What you might consider doing is masking your email addys with a script (see tips forum) or using this guys little gizmos :-)
http://assmaker.mybravenet.com/
http://assmaker.mybravenet.com/aspjmaker.htm
Both are "gorilla approved" btw
4. Yup
My colleagues Thor Knudsen and Cherubino Di Marco in the "gorilla group" posted a long list of "evil bots" that are good to ban using .htaccess
Here's the link:
http://www.frontpagewebmaster.com/m_151710/mpage_3/key_/tm.htm
You'd do well to read that whole thing actually. But that's the key page. Scroll down a bit they gave a lot of info and many good links.
Hope all that helped a little.
TTFN
Fiachra.
| |
|
|
|
Peppergal
Posts: 2207
Joined: 9/20/2002
Status: offline
|
RE: Robots.txt or Robots META tag? - 9/22/2003 16:21:30
Thanks! I am actually writing a robots.txt file right now. I want to know...can I just publish it with FP when I publish other changes I'm making to the site? I'm in the process right now of making an external style sheet for fonts and other attributes as well, so I'm really choking a lot down in a short period of time. My site owner wants to get this site marketed right away and I want to try and clean it up as much as I can (I was reading the SEO thread and the comment made about cleanign up after designers, hee hee) and I don't want to cost my client any more money. I can't do pure CSS right now as the site heavily depends on tables.
So I want to save myself as many steps as I can. I can publish a txt file that was made in Notepad and saved into the directory with the rest of the web pages, correct?
Thanks again!!!
_____________________________
Northeast PA / Poconos/ Lake Wallenpaupack Real Estate
wallenpaupacklakeproperty.com
Karen's Real Estate Blog
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
