|
| |
|
|
Peter
Posts: 133
From: Edinburgh
Status: offline
 |
What page is the Robot looking for? - 5/15/2002 18:27:59
My web site host gives me lots of useful info about the usage of my web site, including pages visited and pages not found.
Most of the 'pages not found' are because I have changed some of the page names, and the search engines have not yet caught up on what I have done - but I keep getting a message 'page not found' for the page:
www.edinphoto.org.uk/robot.txt.
It is true. No such page exists. All the pages on my site are htm, jpg or gif. But should I have also set up a robot.txt page to help the robots, and if so, what info should I include on it?
- Peter
|
|
|
|
caywind
Posts: 1479
From: USA
Status: offline
|
RE: What page is the Robot looking for? - 5/15/2002 19:32:11
robot.txt is a file that you can put in your web site that tells the search engines not to index the list of files that are in the file (robot.txt). what did I just say? anyway don't worry about it, it's not important unless you use it.
Fight the Spam! http://awebmasters.net/antispam.htm
| |
|
|
|
Peter
Posts: 133
From: Edinburgh
Status: offline
|
RE: What page is the Robot looking for? - 5/15/2002 19:55:09
Thanks, caywind.
I get the general idea. I'm happy for the robots to list any of the files they find on my site, so I'll not set up anything under robot.txt
(That's one thing less to think about! I find OutFront to be very helpful in providing quick answers to subjects like this where I have not been able to find the answer in my FrontPage guide books.)
Thank you.
- Peter
| |
|
|
|
Reflect
Posts: 4769
From: USA
Status: offline
|
RE: What page is the Robot looking for? - 5/16/2002 7:24:47
If you just want to make the 404 go away create a robots.txt with notepad. Just open a blank notepad and do a save naming it robots.txt. Import it into the web on your hard drive. Then next time you publish it will go out into the root of your web.
Brian
Work hard, play fair, stay sane
| |
|
|
|
Sandy Fairservice
Posts: 10
Joined: 3/11/2002
From: Christchurch Canterbury New Zealand
Status: offline
|
RE: What page is the Robot looking for? - 5/28/2002 0:06:20
A friend told me to put this bit of HTML in the header. I did, but I don't yet know if it worked. For what it's worth:
<meta name="robots" content="noindex">
Sandy Fairservice
| |
|
|
|
Peter
Posts: 133
From: Edinburgh
Status: offline
|
RE: What page is the Robot looking for? - 5/28/2002 17:43:40
I've got what I need now, thanks to your help.
I'm happy for the robots to see my site. The only thing that was concerning me was the log of pages not found, which was giving higher numbers than I felt comfortable with - but I set up a blank page on notepad as you suggested.
The result: The robots are still looking at my site (as I want them to) and I am not now being told of lots of pages not found.
Thanks for your help.
- Peter
| |
|
|
|
caywind
Posts: 1479
From: USA
Status: offline
|
RE: What page is the Robot looking for? - 5/28/2002 21:26:46
Always glad to be of service.
Fight the Spam! http://awebmasters.net/antispam.htm
| |
|
|
|
Scotty
Posts: 206
From: The left coast-go Obama-
Status: offline
|
RE: What page is the Robot looking for? - 5/29/2002 1:04:21
Wait a minute here! Did I miss something? Peter, I read your thread to say that the search engines are returning lots of errors, (due in part to your changing page names). Don't you want to redirect those searches to you home page or the page they were replaced with? That way you should get the benefit of searches for old and new. Several of the pros dealt with "redirects" on an earlier query of mine. When I find it, I'll list it here. Regards, Scotty
| |
|
|
|
Reflect
Posts: 4769
From: USA
Status: offline
|
RE: What page is the Robot looking for? - 5/29/2002 8:13:17
Hi,
I would actualy make a sitemap. Then calll the host and have them set the sitemap as your default 404 page. That way if a spider/SE comes a knocking for a page that no longer exists they are fed some good and healthy spider food. If you go this root make sure to make the links straight text links (read no Javascript). Also give a good one sentence description using the main words/keywords from the target page.
Brian
Work hard, play fair, stay sane
| |
|
|
|
Peter
Posts: 133
From: Edinburgh
Status: offline
|
RE: What page is the Robot looking for? - 5/29/2002 14:53:43
Reply to Reflect:
Setting up a different 404 page. That sounds like a great idea, thanks. I'll speak to the web site host and find out how to do that.
I may create a page consisting of a brief note saying that I've changed the page names, followed by an extract from the top of my www.edinphoto.org.uk page.
- Peter
| |
|
|
|
Texjd
Posts: 123
From: Houston, Texas
Status: offline
|
RE: What page is the Robot looking for? - 5/29/2002 16:58:44
Standard Operating Procedure: Always make a custom 404 page to return to site map or main menu. If people mistype a url or old or outdated links always will be handled. If you do a search in the root of your web site you can find the existing no info 404 and replace it.
Wasn't there going to be a FAQ somewhere on all the basics of creating a web site? I thought at one time somebody brought this up. This would be a good item to add since it's often overlooked.
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
