Robots.txt Configuration

September 1, 2017

Basic

Allow all bots.

User-agent: *
Disallow:

Block all bots.

User-agent: *
Disallow: /

Prevent all bots from crawling certain directories and files.

User-agent: *
Disallow: /admin/
Disallow: /secret/
Disallow: /member.html

Advance

Disallow all bots except a few.

User-agent: Googlebot
User-agent: Slurp
User-Agent: msnbot
Disallow:

User-agent: *
Disallow: /

Allow all bots except a few.

User-agent: *
Disallow: 

User-agent: BadBot
User-agent: MeanBot
Disallow: /

Allow specific files within a path.

User-agent: Googlebot 
Disallow: /private/ 
Allow: /private/index.html
This work is licensed under a
Creative Commons Attribution-NonCommercial 4.0 International License.