Robots.txt Configuration
September 1, 2017A robots.txt file lives at the root of your site. So, for site
www.example.com, the robots.txt file lives atwww.example.com/robots.txt.
Basic
Allow all bots.
User-agent: *
Disallow:
Block all bots.
User-agent: *
Disallow: /
Prevent all bots from crawling certain directories and files.
User-agent: *
Disallow: /admin/
Disallow: /secret/
Disallow: /member.html
Advance
Disallow all bots except a few.
User-agent: Googlebot
User-agent: Slurp
User-Agent: msnbot
Disallow:
User-agent: *
Disallow: /
Allow all bots except a few.
User-agent: *
Disallow:
User-agent: BadBot
User-agent: MeanBot
Disallow: /
Allow specific files within a path.
User-agent: Googlebot
Disallow: /private/
Allow: /private/index.html
References:
- algolia
- analytics
- android
- android-ktx
- android-permission
- android-studio
- apps-script
- bash
- bootstrap
- bootstrapvue
- chartjs
- chrome
- cloud-functions
- coding-interview
- coroutines
- crashlytics
- css
- dagger2
- datastore
- datetime
- docker
- eslint
- firebase
- firebase-auth
- firebase-hosting
- firestore
- firestore-security-rules
- flask
- fontawesome
- fresco
- git
- github
- glide
- godot
- google-app-engine
- google-cloud-storage
- google-colab
- google-drive
- google-maps
- google-places
- google-play
- google-sheets
- gradle
- html
- hugo
- inkscape
- java
- java-time
- javascript
- jetson-nano
- kotlin
- layout
- lets-encrypt
- lifecycle
- linux
- logging
- lubuntu
- markdown
- mate
- material-design
- matplotlib
- md5
- mongodb
- moshi
- mplfinance
- mysql
- navigation
- nginx
- nodejs
- npm
- nuxtjs
- nvm
- pandas
- payment
- pip
- pwa
- pyenv
- python
- recylerview
- regex
- room
- rxjava
- scoped-storage
- selenium
- social-media
- ssh
- ssl
- static-site-generator
- static-website-hosting
- sublime-text
- ubuntu
- unit-test
- uwsgi
- viewmodel
- viewpager2
- virtualbox
- vue-chartjs
- vue-cli
- vue-router
- vuejs
- vuelidate
- vuepress
- web-development
- web-hosting
- webpack
- windows
- workmanager
- wsl
- yarn