Robots.txt Configuration
September 1, 2017Basic
Allow all bots.
User-agent: *
Disallow:Block all bots.
User-agent: *
Disallow: /Prevent all bots from crawling certain directories and files.
User-agent: *
Disallow: /admin/
Disallow: /secret/
Disallow: /member.htmlAdvance
Disallow all bots except a few.
User-agent: Googlebot
User-agent: Slurp
User-Agent: msnbot
Disallow:
User-agent: *
Disallow: /Allow all bots except a few.
User-agent: *
Disallow:
User-agent: BadBot
User-agent: MeanBot
Disallow: /Allow specific files within a path.
User-agent: Googlebot
Disallow: /private/
Allow: /private/index.html- android
- android-ktx
- android-studio
- bootstrap
- coroutines
- crashlytics
- css
- dagger2
- datastore
- eslint
- firebase
- flask
- fontawesome
- glide
- google-app-engine
- google-drive
- google-maps
- google-places
- google-play
- hugo
- java
- java-time
- javascript
- kotlin
- lets-encrypt
- linux
- markdown
- moshi
- nginx
- npm
- pip
- python
- room
- rxjava
- selenium
- ssl
- static-site-generator
- ubuntu
- unit-test
- uwsgi
- vue-cli
- vuejs
- web-development
- web-hosting
- webpack
- windows
- workmanager