Block all bots/crawlers/spiders for a special directory with htaccess

by

You need to have mod_rewrite enabled. Placed it in .htaccess in that folder. If placed elsewhere (e.g. parent folder) then RewriteRule pattern need to be slightly modified to include that folder name).

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]
  1. I have entered only few bots — you add any other yourself (letter case does not matter).
  2. This rule will respond with “403 Access Forbidden” result code for such requests. You can change to another response HTTP code if you really want (403 is most appropriate here considering your requirements).

More Related Contents:

  1. Redirect non-www to www in .htaccess
  2. How do search engines deal with AngularJS applications?
  3. .htaccess mod_rewrite – how to exclude directory from rewrite rule
  4. Deny all, allow only one IP through htaccess
  5. Password protect a specific URL
  6. How to write htaccess rewrite rule for seo friendly url
  7. symfony2 rewrite rules .htaccess app.php
  8. What exactly does the Multiviews options in .htaccess?
  9. htaccess rewrite breaks relative paths
  10. How to get MAMP to read .htaccess files
  11. Add Trailing Slash .htaccess
  12. .htaccess Remove WWW from URL + Directories
  13. .htaccess 301 redirect of single page
  14. How can I redirect to a different domain without changing the URL in the address bar?
  15. Issue In Removing Double Or More Slashes From URL By .htaccess
  16. Best Practice: 301 Redirect HTTP to HTTPS (Standard Domain)
  17. ‘hash’ url rewrite in .htaccess
  18. htaccess redirect if URL contains a certain string
  19. redirect all .html extensions to .php
  20. Do Google or other search engines execute JavaScript?
  21. .htaccess URL redirect
  22. .htaccess redirect without changing address bar
  23. How do I write a .htaccess file to make CodeIgniters URL routing work?
  24. 301 redirect .htaccess
  25. 301 redirect from URL with query string to new domain with different query string
  26. Preserve HTTP/HTTPS protocol in .htaccess redirects
  27. .htaccess rewrite URL with a question mark “?”
  28. Redirect all http AND https non-www URLS to https://www.xyz.com via htaccess
  29. .htaccess redirect to all IP’s but mine
  30. How can I fix the ‘Missing Cross-Origin Resource Sharing (CORS) Response Header’ webfont issue?

Leave a Comment