1. Tutorials
  2. Web Development

Robots.txt Configuration

September 1, 2017

A robots.txt file lives at the root of your site. So, for site www.example.com, the robots.txt file lives at www.example.com/robots.txt.

Basic

Allow all bots.

User-agent: *
Disallow:

Block all bots.

User-agent: *
Disallow: /

Prevent all bots from crawling certain directories and files.

User-agent: *
Disallow: /admin/
Disallow: /secret/
Disallow: /member.html

Advance

Disallow all bots except a few.

User-agent: Googlebot
User-agent: Slurp
User-Agent: msnbot
Disallow:

User-agent: *
Disallow: /

Allow all bots except a few.

User-agent: *
Disallow:

User-agent: BadBot
User-agent: MeanBot
Disallow: /

Allow specific files within a path.

User-agent: Googlebot
Disallow: /private/
Allow: /private/index.html

References:

❤️ Is this article helpful?
Buy me a coffee☕ or support my work to keep this space 🖖 and ad-free.
If you can't, do send some 💖 to @d_luaz or help to share this article.
👶 Apps I built
Travelopy - discover travel places in Malaysia, Singapore, Taiwan, Japan.
Pixtory App (Alpha) - easily organize photos on your phone into a blog.
暖心芽 (WIP) 🌞❤️🌱 - reminder of hope, warmth, thoughts and feelings (or just quotes).
LuaPass - offline password manager
By Desmond Lua
A dream boy who enjoys making apps, travelling and gardening. Follow me on @d_luaz.
This work is licensed under a
Creative Commons Attribution-NonCommercial 4.0 International License.