Use this tool to generate a simple robots.txt file for your
website. This file allows you to hide files or directories that
you don’t wish the search engine spiders to find. This
generator tool is designed to create the text for the file for
you; you can then make changes afterward should you deem it
necessary. Generating a robot file with
this tool is ideal if you wish to block certain directories
or files from search engines.
To use the generator tool, enter the required information
below and click the button. You will then be shown the text
for the file. Copy this to a file called robots.txt and place
on the root of your website (in the same place as your home
page).
Imposing Restrictions
You may impose restrictions on which web pages to disallow
indexing. By default, most users will want to allow all directories
except their /cgi-bin/ directory, which commonly holds scripts,
and their images directory /images/. To enable all web pages,
select Yes to "Enable All Webpages," then enter
each web page or directory path in the exclusion box, one
per line.
Example: "http://www.yourdomain.com/cgi-bin/"
(Excludes the /cgi-bin/ directory)
Example: "http://www.yourdomain.com/images/" (Excludes
the /images/ directory)
Example: "http://www.yourdomain.com/welcome.html"
(Excludes the /welcome.html web page)
|
