Use a Robots.txt file
Posted: Sun Jan 31, 2010 4:31 pm
Use a Robots.txt file or tags to control how MSNBot and other web crawlers index your website. You can use the Robots.txt file to prevent web crawlers from crawling specific files and folders.