how do web crawlers handle javascript

JavaScript is handled by both Bing and Google crawlers. Yahoo uses the Bing crawler data, so it should be handled as well. I didn’t look into other search engines, so if you care about them, you should look them up.

Bing published guidance in March 2014 as to how to create JavaScript-based websites that work with their crawler (mostly related to pushState) that are good practices in general:

Google later published guidance in May 2014 as to how to create JavaScript-based websites that work with their crawler, and their recommendations are also recommended:

  • Don’t block the JavaScript (and CSS) in the robots.txt file.
  • Make sure you can handle the load of the crawlers.
  • It’s a good idea to support browsers and crawlers that can’t handle (or users and organizations that won’t allow) JavaScript
  • Tricky JavaScript that relies on arcane or specific features of the language might not work with the crawlers.
  • If your JavaScript removes content from the page, it might not get indexed.
    around.

Leave a Comment