Robots.txt: Is this wildcard rule valid?
The answer is, “it depends”. The robots.txt “standard” as defined at robotstxt.org is the minimum that bots are expected to support. Googlebot, MSNbot, and Yahoo Slurp support some common extensions, and there’s really no telling what other bots support. Some say what they support and others don’t. In general, you can expect the major search … Read more