|
| |
|
|
Larry M.
Posts: 2704
Joined: 2/20/2003
From: Greenville, South Carolina, USA
Status: offline
 |
Coding for No Search Engine Return - 2/17/2006 15:48:34
Hello Everyone,
I have 50 or so "private" pages on my real estate site server which are in place (without any meta keywords, except for the title tag) for my Buyer Clients only. Access is by and only known to the respective individual Buyer Client, e.g., www.carolinahomesrealty.com/jones.htm.
I want to positively prevent search engines from crawling or otherwise identifying these pages - my concern is Client privacy won't be protected if one or more pages are picked up by search engines.
With that said, does anyone know of a script or code I can add to the head section of each page that will stop any search engine bots (probably not the correct term)?
Thanks in advance 
_____________________________
Larry M.
Indecision is the Key to Flexibility.
|
|
|
|
Mojo
Posts: 2431
From: Chicago
Status: offline
|
RE: Coding for No Search Engine Return - 2/17/2006 16:10:39
Spooky sells a login program - that's probably your best/easiest solution - unless you can code a simple one yourself. Which, it sounds like all you need is a simple login.
If all you need is to just protect the page I/we can write something simple for you in about 5 minutes.
_____________________________
Split Testing
Chicago Order Fulfillment
Emergency Kits
| |
|
|
|
Mojo
Posts: 2431
From: Chicago
Status: offline
|
RE: Coding for No Search Engine Return - 2/17/2006 16:26:37
quote:
I want to positively prevent search engines from crawling or otherwise identifying these pages
_____________________________
Split Testing
Chicago Order Fulfillment
Emergency Kits
| |
|
|
|
Larry M.
Posts: 2704
Joined: 2/20/2003
From: Greenville, South Carolina, USA
Status: offline
|
RE: Coding for No Search Engine Return - 2/17/2006 18:09:44
quote:
<meta name="robots" content="noindex, nofollow" /> along with a robots.txt file should do it
Womble,
Sounds like what I'm after. However, what is a robots.txt file, how do I build it and where is it inserted when I get one?
Mojo, at the moment I'm going to try and keep it dirt simple but, if that doesn't work, I'll go with Spooky's Login.
_____________________________
Larry M.
Indecision is the Key to Flexibility.
| |
|
|
|
BobbyDouglas
Posts: 5455
Joined: 5/15/2003
From: Arizona
Status: offline
|
RE: Coding for No Search Engine Return - 2/17/2006 18:27:38
quote:
ORIGINAL: Mojo
Just so you know - the meta solution will NOT do what you asked.
- I've noticed this too. Sometimes your website will get listed in SEs despite the fact you have a robots.txt file saying not to index anything at all. You have to either wait for the SE to come around and see the issue, or request your URL to be removed.
_____________________________
Arizona Web Design - Mr Bobs Web Design in Arizona
The Arizona Web Hosting Challenge
| |
|
|
|
Mojo
Posts: 2431
From: Chicago
Status: offline
|
RE: Coding for No Search Engine Return - 2/17/2006 18:33:22
No to mention the rogue search bots that ignore robots.txt (there is no law saying they must obey them anyhow) and the scraper bots. Once a scraper has your content it will be uploaded to pages that the owner *ensures* will be indexed. So, your content will be openly available.
A 5 minute, simple login script (free) will eliminate that problem.
_____________________________
Split Testing
Chicago Order Fulfillment
Emergency Kits
| |
|
|
|
Kitka
Posts: 2513
Joined: 1/31/2002
From: Australia
Status: offline
|
RE: Coding for No Search Engine Return - 2/17/2006 20:10:24
I wouldn't go putting any hidden pages in a robots.txt file!
If the pages are not linked to any pages currently accessed by SE bots, placing the hidden file names in a robots.txt only makes them 100% findable and accessible. If you have ensured that there are no links anywhere for SE bots to follow, it'd be pretty amazingly lucky for them to find them accidently.
Anyone can view a robots.txt file, to look at the one for this site type in:
http://www.frontpagewebmaster.com/robots.txt
... and you'll be able to see exactly which files they don't want seen, or at least listed in the "good" SEs. Bad bots will either ignore a robots.txt or use it to instantly and easily find the files you don't want them to find.
I'd go with using the meta tag womble suggested and/or a simple login as mojo suggested.
_____________________________
Kitka
**It is impossible to make anything foolproof because fools are so ingenious.**
| |
|
|
|
Larry M.
Posts: 2704
Joined: 2/20/2003
From: Greenville, South Carolina, USA
Status: offline
|
RE: Coding for No Search Engine Return - 2/17/2006 21:05:45
quote:
I'd go with using the meta tag womble suggested and/or a simple login as mojo suggested
Kitka,
A trifle confused here. MoJo says, to paraphrase, "the meta solution ... won't work". If there's a free login in script that everyone agrees will work, would you or MoJo (or anyone else) point me in the right direction, please?
Thanks!
_____________________________
Larry M.
Indecision is the Key to Flexibility.
| |
|
|
|
Kitka
Posts: 2513
Joined: 1/31/2002
From: Australia
Status: offline
|
RE: Coding for No Search Engine Return - 2/17/2006 21:13:10
Is the site hosted on Windows or Linux server?
_____________________________
Kitka
**It is impossible to make anything foolproof because fools are so ingenious.**
| |
|
|
|
coreybryant
Posts: 2422
Joined: 3/17/2002
From: Castle Rock CO USA
Status: offline
|
RE: Coding for No Search Engine Return - 2/18/2006 9:13:08
<meta name="robots" content="noindex, nofollow" /> will not work usually for bad robots, but good robots will
Using a robots.txt - you can actually see the harm in doing that. Bad robots won't follow it and you have actually just told the entire world where your hidden files are
If you are on a UNIX server, htaccess would be one way. On IIS, there are a lot of scripts - and sometimes your control panel might even have a function. We have been working on a couple of free scripts as well - just takes time trying to please everyone :) - but you can also check out Spooky Login
_____________________________
Corey R. Bryant
Merchant Accounts | Toll Free Numbers | My Blog | Expression Web Blog
| |
|
|
|
Reflect
Posts: 4769
From: USA
Status: offline
|
RE: Coding for No Search Engine Return - 2/18/2006 11:40:27
I agree, if you REALLY want it private user a response login script as mentioned is the only solution in my mind.
Take care,
Brian
_____________________________
| |
|
|
|
Kitka
Posts: 2513
Joined: 1/31/2002
From: Australia
Status: offline
|
RE: Coding for No Search Engine Return - 2/18/2006 19:00:54
quote:
it sounds like I may be Spooky's next customer.
Spooky Login only works on windoze servers, that is why I asked. It requires ASP.
_____________________________
Kitka
**It is impossible to make anything foolproof because fools are so ingenious.**
| |
|
|
|
Mojo
Posts: 2431
From: Chicago
Status: offline
|
RE: Coding for No Search Engine Return - 2/18/2006 21:08:15
quote:
Was hoping get it done with a simple line of "no spidering" code.
On windows (and I'm sure Unix), it can be done with a simple line - well, maybe 10 of them.
_____________________________
Split Testing
Chicago Order Fulfillment
Emergency Kits
| |
|
|
|
BobbyDouglas
Posts: 5455
Joined: 5/15/2003
From: Arizona
Status: offline
|
RE: Coding for No Search Engine Return - 2/19/2006 1:00:06
quote:
Right now the site is on an Apache Unix server.
- Just put some .htaccess password protection on it with the name of the protected area as "Please enter the number 1 for the username and password" and then have the user/pass be 1 to access this area. Usually this can be setup in your control panel.
quote:
Because Microsoft is no longer supporting Unix with security patches
- Care to explain a bit?
_____________________________
Arizona Web Design - Mr Bobs Web Design in Arizona
The Arizona Web Hosting Challenge
| |
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts
|
|
|
Printable Version 
