Ignore URLs in robot.txt with specific parameters?

by

Here’s a solutions if you want to disallow query strings:

Disallow: /*?*

or if you want to be more precise on your query string:

Disallow: /*?dir=*&order=*&p=*

You can also add to the robots.txt which url to allow

Allow: /new-printer$

The $ will make sure only the /new-printer will be allowed.

More info:

http://code.google.com/web/controlcrawlindex/docs/robots_txt.html

http://sanzon.wordpress.com/2008/04/29/advanced-usage-of-robotstxt-w-querystrings/

More Related Contents:

  1. Robots.txt: Is this wildcard rule valid?
  2. Google search results site map?
  3. Google displaying website title differently in search results
  4. Do Google or other search engines execute JavaScript?
  5. Do SEO-friendly URLs really affect a page’s ranking? [closed]
  6. Angular2 – SEO – how to manipulate the meta description
  7. How does Stack Overflow generate its SEO-friendly URLs?
  8. When should I use a trailing slash in my URL?
  9. JSON-LD Schema.org: Multiple video/image page
  10. What is the purpose of the “role” attribute in HTML?
  11. What is the etymology of ‘slug’ in a URL? [closed]
  12. Generate SEO friendly URLs (slugs) [closed]
  13. Replacing H1 text with a logo image: best method for SEO and accessibility?
  14. Remove index.php?route=common/home from OpenCart
  15. What happens if the meta tags are present in the document body?
  16. Google bot crawling on AngularJS site with HTML5 Mode routes
  17. “Single-page” JS websites and SEO
  18. Why do some websites add “Slugs” to the end of URLs? [closed]
  19. Doing links like Twitter, Hash-Bang #! URL’s [duplicate]
  20. Can I Use Multiple ItemProps in a Span Tag for schema.org Rich Snippets?
  21. Google SEO and hidden elements
  22. Is text-indent: -9999px a bad technique for replacing text with images, and what are the alternatives?
  23. HTML5 i tag validity with icons
  24. Best JSON-LD practices: using multiple elements?
  25. How to properly use h1 in HTML5
  26. Stop Google from indexing [closed]
  27. ASP.net MVC support for URL’s with hyphens
  28. Does Google crawl AJAX content? [closed]
  29. Is it possible to do friendly url (url rewriting) in classic asp?
  30. URLs: Dash vs. Underscore [closed]

Leave a Comment