How to overcome an HTMLUnit ScriptException?

by

HtmlUnit does not play well with JavaScript. It will frequently throw errors complaining about variables or functions not defined.

In that sense, real life browsers (FireFox, Internet Explorer, Chrome, etc) are much more flexible. That means that they will allow syntactically incorrect pieces of HTML and JavaScript (eg: not defining functions or not ending HTML tags).

HtmlUnit expects everything to be (almost) perfect. Although, it will fix some missing ending HTML tags, in general, it expects the code in the pages not to contain any kind of error. Furthermore, even if everything looks correct HtmlUnit might even complain.

Some items for you to think about are:

  • The most important one is switching between different BrowserVersions. You can set them when creating the WebClient object. Internet Explorer (ironically) has proven to give me the best results when it comes to interpreting JavaScript
  • Make sure your HTML and JavaScript code are both correct
  • Avoid using of complex libraries (jQuery seems to be properly supported)
  • Try to use non-minimized versions of libraries
  • If you happen to be using jQuery (or other similar libraries) avoid complex jQuery methods (eg: dynamically adding events to elements)

Of course, those comments would apply if you have control over the source code you’re fetching from the server. Sometimes, this is not the case. In this situation your hands are even more tied.

One option, would be to suppress the exception with:

webClient.getOptions().setThrowExceptionOnScriptError(false);

Although, this will get you through the exception won’t correct any JavaScript error. That means that if the JS piece of code that is throwing this exception happens to be crucial in your logic, I mean, you absolutely depend on the result of the execution of that code then you can not let HtmlUnit handle your JS. If this happens to be the result of an AJAX request then you can issue the request yourself manually instead of letting HtmlUnit do so.

On the other hand, if the JS code that is giving you trouble is not critical in your logic, I mean, it might just be hiding an element or changing a color that you don’t care about, then suppressing the exception would be the way to go.

There aren’t many options left.

More Related Contents:

  1. Getting Jsoup to support dynamically generated html by JavaScript
  2. Dynamic Drop down values based on another dropdown [closed]
  3. How to use JavaScript with Selenium WebDriver Java
  4. Spring JSON request getting 406 (not Acceptable)
  5. Java equivalent to JavaScript’s encodeURIComponent that produces identical output?
  6. calling java methods in javascript code [closed]
  7. Get the changed HTML content after it’s updated by Javascript? (htmlunit)
  8. How can I use JavaScript in Java? [closed]
  9. Execute managebean method from javascript onload event
  10. How can I get stock quotes using Google Finance API?
  11. Android Web Scraping with a Headless Browser [closed]
  12. Why does changing the sum order returns a different result?
  13. Jsoup get dynamically generated HTML
  14. Html/Javascript debugging in JavaFX WebView
  15. Detecting Device Type in a web application
  16. sending variable from one jsp to another jsp
  17. How can I show/hide component with JSF?
  18. How to use HTML and CSS as a Java application GUI?
  19. Assign JavaScript variable to Java Variable in JSP
  20. How to type some text in hidden field in Selenium WebDriver using Java
  21. How to check whether a port is open at client’s network/firewall?
  22. Convert Javascript regular expression to Java syntax
  23. calling java method in javascript [closed]
  24. Why is JavaScript called JavaScript, since it has nothing to do with Java? [closed]
  25. I get a status 200 when connecting to the websocket, but it is an error?
  26. Understanding Android’s webview addjavascriptinterface
  27. Detect version of Java using JavaScript
  28. How to differentiate Ajax requests from normal Http requests?
  29. org.openqa.selenium.UnhandledAlertException: unexpected alert open
  30. Android WebView always returns null for javascript getElementById on loadUrl

Leave a Comment