Robots.txt Configuration
September 1, 2017A robots.txt file lives at the root of your site. So, for site
www.example.com, the robots.txt file lives atwww.example.com/robots.txt.
Basic
Allow all bots.
User-agent: *
Disallow:Block all bots.
User-agent: *
Disallow: /Prevent all bots from crawling certain directories and files.
User-agent: *
Disallow: /admin/
Disallow: /secret/
Disallow: /member.htmlAdvance
Disallow all bots except a few.
User-agent: Googlebot
User-agent: Slurp
User-Agent: msnbot
Disallow:
User-agent: *
Disallow: /Allow all bots except a few.
User-agent: *
Disallow:
User-agent: BadBot
User-agent: MeanBot
Disallow: /Allow specific files within a path.
User-agent: Googlebot
Disallow: /private/
Allow: /private/index.htmlReferences:
- android
- android-ktx
- android-studio
- bootstrap
- cloud-functions
- coding-interview
- coroutines
- crashlytics
- css
- dagger2
- datastore
- eslint
- firebase
- firebase-auth
- firestore
- flask
- fontawesome
- git
- glide
- google-app-engine
- google-drive
- google-maps
- google-places
- google-play
- google-sheets
- gradle
- html
- hugo
- java
- java-time
- javascript
- kotlin
- lets-encrypt
- lifecycle
- linux
- logging
- markdown
- md5
- moshi
- nginx
- npm
- payment
- pip
- python
- recylerview
- regex
- room
- rxjava
- selenium
- ssl
- static-site-generator
- sublime-text
- ubuntu
- unit-test
- uwsgi
- virtualbox
- vue-cli
- vuejs
- vuepress
- web-development
- web-hosting
- webpack
- windows
- workmanager