A robots.txt file lives at the root of your site. So, for site
www.example.com, the robots.txt file lives atwww.example.com/robots.txt.
Basic
Allow all bots.
User-agent: *
Disallow:Block all bots.
User-agent: *
Disallow: /Prevent all bots from crawling certain directories and files.
User-agent: *
Disallow: /admin/
Disallow: /secret/
Disallow: /member.htmlAdvance
Disallow all bots except a few.
User-agent: Googlebot
User-agent: Slurp
User-Agent: msnbot
Disallow:
User-agent: *
Disallow: /Allow all bots except a few.
User-agent: *
Disallow:
User-agent: BadBot
User-agent: MeanBot
Disallow: /Allow specific files within a path.
User-agent: Googlebot
Disallow: /private/
Allow: /private/index.htmlReferences: