Java Glossary : robots.txt

Last updated 2004-06-28 by Roedy Green ©1996-2004 Canadian Mind Products

Java definitions: 0-9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

You are here : home : Java Glossary : R words : robots.txt.

robots.txt

robots.txt is a file you can place in the root directory of your website to tell web crawlers (search engines) which pages to index and which to ignore. A typical robots.txt file might look like this:

# parts of the mindprod.com website not indexed
user-agent: *
disallow: /template.html
disallow: /mindprod/include
disallow: /mindprod/jgloss/include
disallow: /mindprod/addictions

It means, for all browsers, don't look at the file template.html or anything in the three directories mentioned. There is no way to tell it to avoid certain file extensions.

home	Canadian Mind Products
	mindprod.com IP:[24.87.56.253]
	Your IP:[80.134.30.163]
	You are visitor number 654.
	Please send errors, omissions and suggestions
	to improve this page to Roedy Green.
	You can get a fresh copy of this page from:	or possibly from your local J: drive mirror:
	http://mindprod.com/jgloss/robotstxt.html	J:\mindprod\jgloss\robotstxt.html