|
| |
|
|
Mike_R
Posts: 124
Joined: 1/15/2003
Status: offline
 |
RE: Google Page Rank and Search Results Position? - 9/16/2003 14:46:30
I see. I like the idea--it make's sense. As Brian suggests, I may be too late. I wish I had known in the beginning. Oh well, I guess I will have to design another site someday.
Incidentally, do you know why only the first page in a section/folder gets ranked by Google? I was curious about that. And, does it harm or affect your ranking on other pages when pages like your privacy policy get indexed?
Mike
|
|
|
|
Mike_R
Posts: 124
Joined: 1/15/2003
Status: offline
|
RE: Google Page Rank and Search Results Position? - 9/16/2003 19:03:42
Brian, I didn't take it as a slam at all. I know that in the learning process of developing my first site, I made some bad calls in defining priorities. I smile at it now and actually feel proud that I have learned as much as I have. Nothing like a good screw-up to teach a good, and permanent lesson. The next time I build a site, I am going to be able to do it in 1/3 the time I built this one. And I owe most of that to the expert advice received from you and others in the Outfront Forums. You guys have really saved my sanity a few times.
Mike
| |
|
|
|
coreybryant
Posts: 2625
Joined: 3/17/2002
From: Castle Rock CO USA
Status: offline
|
RE: Google Page Rank and Search Results Position? - 9/17/2003 11:11:57
Andrew, when you say:
quote:
My motivation is to prevent the bot from becoming lost or timing out before it can get some real data. I also block the FP directories and cgi-bin:
Do the bots actually find some of those web pages if they are not actually linked? I could possibly see the cgi-bin & _private but what about _vti_cnf - since I do not think those folders are actually referenced anywhere in the FP coding?
I added a lot to my robots page (to help keep the spammers & e-mail harvesters out) so I am interested in knowing more about this actually as well. Thank you!
< Message edited by coreybryant -- 9/17/2003 8:12:12 AM >
_____________________________
Corey R. Bryant
My Merchant Account Blog | Expression Web Blog
| |
|
|
|
awasson
Posts: 80
From: Vancouver BC Canada
Status: offline
|
RE: Google Page Rank and Search Results Position? - 9/19/2003 11:14:47
Hi Corey,
Sorry about my lack of response. This silly work thing gets in the way sometimes.
The calendar is linked through my navigation section, under application development services, and the bots would get in there and more than likely time out while indexing thoudands of pages of dynamic content, or expire trying to.
This also meant that in the case of google I would get visited by the google bot about once a week if I was lucky.
Since I blocked the calendar, I get visits almost every day without fail and my results are much better in the ranks.
I also blocked the _vti_cnf and other private folders just to be safe. It just takes a minute. If you're on Unix you can make adjustments that prevent certain user agents access from chosen directories by adjusting your .htaccess file.
I think it would be possible to do the same thing with ASP on a windows server to run a small script at the top of every page that loops through a list and if it finds a known spam agent it dumps them into an empty directory and also redirects good bots to the places you want them.
Cheers,
Andrew
| |
|
|
|
pageoneresults
Posts: 1001
From: Orange, CA USA
Status: offline
|
RE: Google Page Rank and Search Results Position? - 9/22/2003 10:11:40
Any folders in your web that begin with an _ (underscore) are automatically off limits to any spiders and/or users. Typically if you browse to those files with the underscore in them, you will get a username/password prompt so there really is no need to clutter your robots.txt file with them. I always put the _private in there just to be on the safe side, the others are of no importance.
If you are that concerned about filtering or directing PR within your site, you need to give careful consideration to your linking structure. You can easily block certain pages from being indexed by using the robots.txt file or a META Robots tag.
_____________________________
SEO Consultants Directory
Find Search Engine Marketing Companies
| |
|
|
|
Mike_R
Posts: 124
Joined: 1/15/2003
Status: offline
|
RE: Google Page Rank and Search Results Position? - 9/23/2003 23:54:03
quote:
If you are that concerned about filtering or directing PR within your site, you need to give careful consideration to your linking structure. You can easily block certain pages from being indexed by using the robots.txt file or a META Robots tag.
Pageone, you got me thinking with that comment. If I list my "Includes" folder in my robots.txt file, will that keep the spiders from indexing the individual pages (header, footer, menu, sidebar, e.g.), or will it keep the spider from indexing all pages that use those pages--i.e. all of my content pages?
Mike R
| |
|
|
|
Reflect
Posts: 4765
From: USA
Status: offline
|
RE: Google Page Rank and Search Results Position? - 9/24/2003 7:45:47
quote:
If I list my "Includes" folder in my robots.txt file, will that keep the spiders from indexing the individual pages (header, footer, menu, sidebar, e.g.), or will it keep the spider from indexing all pages that use those pages--i.e. all of my content pages?
I believe that it would still read the content as a spider/bot sees the include inline, not as a seperate page (the include has been rendered like you viewing the page with a browser). Now if it is just trying to crawl the directory then I would think it would look at the robots.txt file and bypass.
I say this as I have included my "includes" directory in my robots.txt on my first site. The bots however showed pages in its SERPs that could have only been reached via my "include" pages navigation.
Brian
_____________________________
| |
|
|
|
flintops
Posts: 3
Joined: 2/16/2004
Status: offline
|
RE: Google Page Rank and Search Results Position? - 2/16/2004 14:18:50
Hi guys,
Also remember in in text links too. As the Google technology gets more sophisticated, then links pages will carry less value.
Kind regards,
Paul
< Message edited by Mojojo -- 2/16/2004 18:21:22 >
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
.