|
| |
|
|
futurewebboss
Posts: 44
Joined: 6/9/2007
Status: offline
 |
Indexing Password Protected Pages - 7/22/2007 10:37:21
In general, will a search engine robot index password protected pages?
I have quite a number of pages that require username and password and need to know if these will be indexable by the bots.
Thanks,
|
|
|
|
Nicole
Posts: 2959
Joined: 9/15/2004
From: Sydney, Australia
Status: offline
|
RE: Indexing Password Protected Pages - 7/22/2007 16:32:59
Hi Daniel,
I'm not sure but I do have a friend who found her private contact details through a search engine on a supposedly password protected page.
Why don't you include a robots.txt file or a <meta name="robots" content="index,follow" /> to your head information?
_____________________________
". . . and God created man in his image." errr, shouldn't that be "and man created God in his image'?
| |
|
|
|
futurewebboss
Posts: 44
Joined: 6/9/2007
Status: offline
|
RE: Indexing Password Protected Pages - 7/22/2007 19:19:27
Thanks Nicole,
I certainly thought about that, but still wasn't sure if the robots.txt file would proive access since there is an include file in each of the password protected pages. The problem is that there isn't any manner to verify the robots file to determine if it is actually seeing the password protected pages.
Dan
| |
|
|
|
ou812
Posts: 1705
Joined: 1/5/2002
From: San Diego
Status: offline
|
RE: Indexing Password Protected Pages - 7/22/2007 23:18:55
Non of my password protected pages have made it to any search engines on any of the sites I've done (that I've been able to see). I would think that if they do, the protection isn't really working. If the search engines can get to it, then I would think anyone could, and it wasn't protected after all.
_____________________________
-brian
Black Holes suck.
EnterpriseDB: Enterprise-class relational database management system
PostgreSQL: The world's most advanced open source database
| |
|
|
|
Mojo
Posts: 2472
From: Chicago
Status: offline
|
RE: Indexing Password Protected Pages - 7/23/2007 0:57:17
quote:
If the search engines can get to it, then I would think anyone could, and it wasn't protected after all.
Correct.
The search bots just follow links and read the output. If they run into a password protected page, they will index the message on that page. I find cached password forms on Google all the time.
_____________________________
Split Testing
GPS Vehicle Tracking
Chicago Marketing
| |
|
|
|
Mehdi
Posts: 32
Joined: 8/1/2007
Status: offline
|
RE: Indexing Password Protected Pages - 8/2/2007 9:38:11
If you don't want the pages indexing add a dissalow tag in your robots.txt file e.g.
Disallow: /privatefolder/
| |
|
|
|
futurewebboss
Posts: 44
Joined: 6/9/2007
Status: offline
|
RE: Indexing Password Protected Pages - 8/2/2007 9:41:59
Thanks for all the great replies. This is absolutley the best community on the net.
I know exactly what I"m going to do now.
| |
|
|
|
Mehdi
Posts: 32
Joined: 8/1/2007
Status: offline
|
RE: Indexing Password Protected Pages - 8/7/2007 22:20:00
quote:
ORIGINAL: Mojo
quote:
If the search engines can get to it, then I would think anyone could, and it wasn't protected after all.
Correct.
The search bots just follow links and read the output. If they run into a password protected page, they will index the message on that page. I find cached password forms on Google all the time.
Is this still the case for files and folders disallowed in a robots.txt file?
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 