Robots.txt: Is this wildcard rule valid?

The answer is, “it depends”. The robots.txt “standard” as defined at robotstxt.org is the minimum that bots are expected to support. Googlebot, MSNbot, and Yahoo Slurp support some common extensions, and there’s really no telling what other bots support. Some say what they support and others don’t.

In general, you can expect the major search engine bots to support the wildcards that you’ve written, and the one you have there looks like it will work. Best bet would be to run it past one or more of these robots.txt validators or use Google’s Webmaster tools to check it.

Leave a Comment