Android Web Scraping with a Headless Browser [closed]

by

Ok after 2 weeks I admit defeat and are using a workaround which works great for me at the moment.

The problem:
It is too difficult to port HTMLUnit to Android (or at least with my level of expertise). I am sure its a worthwhile project (and not that time consuming for experienced java programmer) . I emailed the guys at HTMLUnit and they commented that they are not looking into a port or what effort will be involved but suggested anyone who wants to start with such a project should send an message to their mailing list to get more developers involved (http://htmlunit.sourceforge.net/mail-lists.html).

The workaround:
I used android’s built in WebView and overrided the onPageFinished method of Webview class to inject Javascript that grabs all the html after the page has fully loaded. Webview can also be used to called futher javascript actions, clicking buttons, filling in forms etc.

Code: 

webView.getSettings().setJavaScriptEnabled(true);
MyJavaScriptInterface jInterface = new MyJavaScriptInterface();
webView.addJavascriptInterface(jInterface, "HtmlViewer");

webView.setWebViewClient(new WebViewClient() {

    @Override
    public void onPageFinished(WebView view, String url) {
       //Load HTML
       webView.loadUrl("javascript:window.HtmlViewer.showHTML('<html>'+document.getElementsByTagName('html')[0].innerHTML+'</html>');");
    }

}

webView.loadUrl(StartURL);
ParseHtml(jInterface.html);   

public class MyJavaScriptInterface {

    public String html;

    @JavascriptInterface
    public void showHTML(String _html) {
        html = _html;
    }
}

More Related Contents:

  1. An HtmlUnit alternative for android?
  2. How to connect via HTTPS using Jsoup?
  3. Creating new Android (Java ) file in Existing Android package [closed]
  4. I'm trying to create a basic android application on eclipse [closed]
  5. Handler as a method parameter
  6. How to launch an Activity from another Application in Android
  7. Launching Google Maps Directions via an intent on Android
  8. Get nested JSON object with GSON using retrofit
  9. Java 7 language features with Android
  10. How to add a recyclerView inside another recyclerView
  11. How can I print an image on a Bluetooth printer in Android?
  12. Unable to resolve host “” No address associated with host name [closed]
  13. How to set background color of an Activity to white programmatically?
  14. Android Studio – No JVM Installation found
  15. Sort data to RecyclerView based on latest date from Firebase
  16. Error building AAB – Flutter (Android) – Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
  17. creating Hashmap from a JSON String
  18. Android: TextView automatically truncate and replace last 3 char of String
  19. Error:Execution failed for task ‘:app:processDebugResources’. > java.io.IOException: Could not delete folder “” in android studio
  20. How to change size of title’s text on Action Bar?
  21. Android buildscript repositories: jcenter VS mavencentral
  22. Selenium WebDriver and DropDown Boxes
  23. Is Dalvik’s memory model the same as Java’s?
  24. Convert a String to a byte array and then back to the original String
  25. android color between two colors, based on percentage?
  26. Android Screen Timeout
  27. How to set default download directory in selenium Chrome Capabilities?
  28. Multiple dex files define /BuildConfig, can’t find the cause:
  29. How to perform Basic Authentication for FirefoxDriver, ChromeDriver and IEdriver in Selenium WebDriver?
  30. Can one class extend two classes?

Leave a Comment