Robots.txt Configuration

September 1, 2017

A robots.txt file lives at the root of your site. So, for site www.example.com, the robots.txt file lives at www.example.com/robots.txt.

Basic

Allow all bots.

User-agent: *
Disallow:

Block all bots.

User-agent: *
Disallow: /

Prevent all bots from crawling certain directories and files.

User-agent: *
Disallow: /admin/
Disallow: /secret/
Disallow: /member.html

Advance

Disallow all bots except a few.

User-agent: Googlebot
User-agent: Slurp
User-Agent: msnbot
Disallow:

User-agent: *
Disallow: /

Allow all bots except a few.

User-agent: *
Disallow:

User-agent: BadBot
User-agent: MeanBot
Disallow: /

Allow specific files within a path.

User-agent: Googlebot
Disallow: /private/
Allow: /private/index.html

References:

This work is licensed under a
Creative Commons Attribution-NonCommercial 4.0 International License.